Factoring has become easier over the last 15 years for two reasons: computer hardware has become more powerful, and better factoring algorithms have been developed.

Hardware improvement will continue inexorably, but it is important to
realize that *hardware improvements make RSA more secure, not less. *This
is because a hardware improvement that allows an attacker to factor a number
two digits longer than before will at the same time allow a legitimate
RSA user to use a key dozens of digits longer than before; a user can choose
a new key a dozen digits longer than the old one without any performance
slowdown, yet a factoring attack will become much more difficult. Therefore,
although the hardware improvement does help the attacker, it helps the
legitimate user much more. This general rule may fail in the sense that
factoring may take place using fast machines of the future, attacking RSA
keys of the past; in this scenario, only the attacker gets the advantage
of the hardware improvement. This consideration argues for using a larger
key size today than one might otherwise consider warranted. It also argues
for replacing one's RSA key with a longer key every few years, in order
to take advantage of the extra security offered by hardware improvements.
This point holds for other public-key systems as well.

Better factoring algorithms have been more help to the RSA attacker than have hardware improvements. As the RSA system and cryptography in general have attracted much attention, so has the factoring problem, and many researchers have found new factoring methods or improved upon others. This has made factoring easier for numbers of any size, irrespective of the speed of the hardware. However, factoring is still a very difficult problem.

Overall, any recent decrease in security due to algorithm improvement can be offset by increasing the key size. In fact, between general computer hardware improvements and special-purpose RSA hardware improvements, increases in key size (maintaining a constant speed of RSA operations) have kept pace or exceeded increases in algorithm efficiency, resulting in no net loss of security. As long as hardware continues to improve at a faster rate than the rate at which the complexity of factoring algorithms decreases, the security of RSA will increase, assuming RSA users regularly increase their key sizes by appropriate amounts. The open question is how much faster factoring algorithms can get; there could be some intrinsic limit to factoring speed, but this limit remains unknown.