At present the side covers on my desktop machine are off, and have been for a few months. Even though I smoke cigars at my computer desk, the interior of the computer is completely clean -- which amazes me. It probably helps that the computer case has filters over the air intakes, and it sits on a small table that keeps it away from the carpet.
I may inspect more carefully and use canned air on the cooling fans and the case's bottom panel (just to make sure) before I put the side panels back; and of course I'll pull, clean and replace the filters although they seldom collect much dust.
With previous computers I would remove the left-side panel every six months or so and dust with canned air, or more often if I was performing an upgrade. Never found more than a slight amount of dust, as long as they stayed off the floor. Some of my friends' computers (like many of us, I'm "tech support") are another story entirely; I've been known to move their computers to other locations, and sometimes leave off a side panel so they can see the dust that collects.