enderwins,
It's a subjective question that is best based on your personal preference.
That said, I always recommend HDTech's Rule of Three. Meaning three times the corner to corner of your widescreen TV is the starting point to best measure where you're going to sit for watching television or movies.
So a 46" television x3 = 138 inches = @ 11 feet.
Abd a 52" television x3 = 156 inches = @ 12 feet.
If you're a gamer and want to get lost in the screen, bigger is better, sure. But what you want to do is sit where you'd normally sit, watch a widescreen movie and relax your eyes. If your peripheral vision can capture the whole screen without you having to move your eyes back and forth, then you're where you want to be - this will deliver a comfortable, relaxing experience.
It sounds crazy, but I'd go to your favorite retailer, measure out the approximate distance, and watch the television from that distance on both the 46" and 52", and concentrate on which experience you enjoy more. Ask yourself if you're easily able to see the entire picture comfortably.
If your eyes are darting to the left and right on various screens, then your panel may be too big for long-term (multiple hours) viewing.
Most people figure that bigger is better, and while it might be for some people, some are looking for something that best fits their needs.
Does that help? Let me know if I can help further. 
--HDTech