It definitely looks like you are having some kind fuzziness or compression problems. I'm curious what the background environment will be (color and brightness).
Anyway, in the camcorder realm (of green screening) there are three things you want to consider. The image sensors, the lens system, and the compression method. And that's probably in order of increased cost.
Having three CCDs in a camera will generally help your color saturation immensely, which has a side effect of helping the compression engine. For the price range, those Panasonic cameras mentioned above are likely as good as you can get.
However the second phase, the lens system, can have serious impact on the crispness of the final image. There's a reason professional lens prices go up into the $50,000+ range. Having precise optical refraction is hard to accomplish. And the cheaper a lens gets the more you can bank on them creating fuzziness, chromatic separation, and plenty of other tiny problems that generally make for a less than perfect image. A couple years ago the Canon GL2 had one of the best lens systems in the integrated camcorder/lens market.
As far as the third option uncompressed is the best you can get. But MiniDV is probably the least destructive in the sub $2k market. I suppose in the HD realm you can get some MPEG2 that matches the MiniDV quality. If you are really wanting to jump up to a better compression system, look for terms like 4:2:2 or 4:4:4 that indicate the data stored from the original 4:4:4 color system coming off the chips. However in the price range you are talking you are going to be limited to the 4:1:1 of MiniDV or 4:2
of Mpeg2 (if I remember correctly).
Considering all of that, how much of a clean mask have you been able to create with your current footage? If your green screen lighting doesn't match the the background plate then it's going to look layered not matter how good the mask is.
Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic