Mac OS X's Desktop is the de facto location for downloaded files, and for many users, in-progress works that will either be organized later or deleted altogether. The desktop can also be gluttonous, however, becoming a catch-all for files that linger indefinitely.
Unfortunately -- aside from the effect of disarray it creates -- keeping dozens or hundreds of files on the Desktop can significantly degrade performance. Not necessarily because the system is sluggish with regard to rendering the icons on the desktop and storing them in memory persistently (which may be true in some cases), but more likely because keeping an excessive number of items on the Desktop can cause the windowserver process to generate reams of logfiles, which obviously draws resources away from other system tasks.
As such, keeping as few items as possible on the Desktop prove a surprisingly effective performance boon. Even creating a single folder on your Desktop and placing all current and future clutter inside, then logging out and back in can provide an immediately noticeable speed boost -- particularly for the Finder.