Most consumer camcorders - including your CX260 - use a technology called AVCHD to compress the video. Many other compression things happen, but the big deal is that frame 1 = the "base". The next 7 frames = what changed from frame 1. Keep in mind that you'll be recording at NTSC standard frame rate of 29.97 frames per second (normally referred to as 30 fps). Your goal is to reduce compression as much as you can.
Less compression = higher quality = bigger file size.
More compression = reduced quality = smaller file size.
If you want "best quality" i.e., best resolution, then you will use PS or FX. Least compression.
There are other factors that will impact the "visual quality":
1) The lens and single imaging chip on your CX260 are small. That means not a lot of light is allowed into the camcorder for the small imaging chip to begin with. While I understand that Volleyball is played indoors under lights, the camcorder and your eyes are very different. This sort of indoor lighting is not very good - for camcorders. Be sure to learn to use the aperture/iris settings.
2) The indoor lights can have a blue or orange tint to them. Be sure to use the white balance setting. Manual is best - presets wok fine, too.
3) Before the "real" recording activity, Practice practice, practice - just like the players. Capture video during practices or - better yet - under the SAME lighting conditions that the game(s) will be played. Import and edit the video in the computer. Burn a DVD. This is the ONLY way you will know if what you bought is capable of providing you with what you want. Use the extended batteries to be sure they work.
Turn digital zoom off. It is useless.
Since a Volleyball game is not 4-6 hours long, you have plenty of recording space. Just get the video into the computer after the match, then you can re-use the SD cards for future events. We don't know which batteries you got, but if you got extended batteries, then you *should* be fine - assuming the batteries are from the camcorder manufacturer, and not third party replacements. SD cards are not "long term archive" media. Hopefully you got the appropriate Class 4 (Class 6 would be preferred) cards that can keep up with all the high definition video data coming in during recording.
Once the video is in the computer - assuming your computer has the horsepower to deal with AVCHD-compressed video and the video editor can deal with the file format, you can make either a computer-readable data file or a standard definition DVD. The stuff on the disc in the box is next to useless. An editor capable of dealing with capturing/importing AVCHD video like Sony Vegas or Adobe Premiers for Windows or iMovie or Final Cut for Macintosh is preferred. Making a DVD requires a DVD authoring tool - like MyDVD for Windows or iDVD for Macintosh. There are many other applications available.
The trick is capturing the video you want at the quality you want. Try not to zoom or pan too much.
When you get the video into the computer, keep in mind that going frame-by-frame is not going to provide you with any measure of the video clarity. You will likely see the image as blurry. What counts is what it looks like at normal playback. Depending on the video editor used, slowing to around 17 frames per second can likely yield useful results, too - if some slow-motion is desired
I just bought a Sony HDR-CX260V camcorder that I intend to use for recording volleyball matches my daughter is playing in. It will be mounted on a tripod and record continuously through each match. I will later edit the video to pull out clips and assemble them to send to a college coaches. I am guessing they will watch them on their computer, but not sure about that yet.... may use a DVD player.
I bought some 64G cards and also have extended batteries.
So looking at the manual, I'm not sure how much benefit I gain in choosing the different quality levels vs. that added battery drain and space usage.
PS - 28Mbps 1920 x1080
FX - 24Mbps 1920 x 1080
FH - 17Mbps 1920 x 1080
HQ - 9Mbps 1440 x1080
LP - 5Mbps 1440 x 1080
and then whether to choose interlaced or progressive.
The default is HQ and claims to go 660-880 minutes on 64G.
Going with FX would cut that down to 360 minutes on a 64G card.
I'm not sure if they increased quality levels are going to dramatically change the output.