I have a friend with a Panasonic 50" Plasma, and I can't say I'm too impressed either. It seems to be very susceptible to noise from the cable box signal -- or that's the best I can make of the prob.
Now, your 32" is better -- well, that's a whole different story (you're comparing apples to oranges). The deal here is that with the 50" you're basically expanding the same amount of vertical lines, so with a larger screen you're more likely to see the incoming signal's imperfections (stretch your 32" to 50" and you'll see garbage there too in non-HTDV signals). The reality with non-HDTV channels is: Bigger screens amplify the intrinsic noise coming out of the cable box and such. I've noticed the big retail chains are now showing their wide/bigsceens in a way to mask this issue (direct satellite feeds with 720p to match native screen resolutions). The surprise is when you plunk down $6K for a plasma display (that will probably be at its best for 2 years) and come home, hook it up to your cable box and see all the noise the cable company and underlying media providers are sending you that you didn't previously discern on smaller size/resolution screens.