Yes. And it is a myth that updates make a computer slow. Still, even if we assumed it to be true, the alternative would be to put your system at ever increasing risk of being compromised and then used for anything from sending out all those viagra spam emails to hosting kiddie porn images that will leave you holding the bag with sex offender charges when the FBI traces it back to your computer. Even if you successfully defend yourself against the charges, you'll have hefty legal bills and people will always whisper about how you were suspected as being a child molester.

Is either of those scenarios, or the hundreds of possibilities in between, preferable to a minor performance loss? If they are, then feel free to ignore updates. You may not use your computer as a server, but rest assured, someone else will be quite happy to do so on your behalf.