There is a difference in the method by which a signal is transmitted over either cable, so there really isn't a way to compare the two, although the RCA cable (composite video signal) is preferred over coax (modulated signal).
If you are experiencing an inferior quality picture with the RCA input, it might be hard to pin down where the problem is. You can start with the cables, but I doubt that is an issue. Some cables are made better than others, but you will not likely see a significant improvement.
It could be the output of the cable box itself. What make and model is it? You may be able to do some research on the web to see if others are experiencing the same problem. Ask your provider if they can provide you with a different model.
Hopefully it isn't you TV, but it could be. There could be a big difference in the way it is processing video from each source. Again, you might find some information through web search for your make and model of TV.
Hope you have some luck.
I have a 32" LG LCD TV and I just got a digital cable box (terminal) from my cable tv provider. There's both a coax cable and RCA cables (3 cables - NOT HD) running from the terminal to the tv. With the remote I can choose between different video sources: "TV" refers to the coax cable; "AV1" and "AV2" refer to 2 sets of RCA ports. If I select "TV" then the picture quality is better than if I pick AV1 or AV2. I also have 2 sets of RCA cables. I tested one set with AV1 and the other set with AV2. Reds, yellows, light greens, oranges, and pinks all look brighter when I change the source to "TV". Is this typical? Is coax better when it comes to picture quality than RCA cables? Since AV1 and AV2 both look inferior to "TV", I think it's unlikely that the problem is the TV. Otherwise, I would have 2 sets of bad RCA jacks on my TV. Does this sound typical or do you think I might have bad cables or a bad digital terminal?
Are all RCA cables the same or are some brands better than others? I could buy a new set of cables and try again.