Devices that do not demand the processing of information while idle will have little effect on the speed of the computer. The best way to find that out would be to use Task Manager and view the Performance tab with each plugged in. As far as starting up and shutting down, there you could see noticeable changes in those times with many devices plugged in. Windows will try to startup or shutdown any processes these devices use. If one of them is reluctant to give up control, it will slow down the startup or shutdown time. BTW: You could use a USB 2.0 hub to bring the connections to a point you can reach instead of on the back of your computer case.
I'm lazy and find it hassle to keep pulling the computer out and plugging/unplugging USB devices.
Can it slow my computer down if I leave them in all the time? What about starting up or shutting down?