A number of people are experiencing poor tracking performance with Apple's new Magic mouse when using it with Mac Pro computers. While the same mice work well with iMacs and laptops, it seems some signal power or interference issue with certain Pro desktops is causing the mouse to have jittery traction.
Software configuration issue?
This problem is intermittent for some, with the mouse working well and then losing its connection after a few minutes. If this is the case, try moving the mouse closer to the computer to test whether you have a weak Bluetooth signal. If the jittery behavior happens regardless of the distance from the computer, it indicates a potential software issue, and you can try several things:
Reapply the latest OS and Magic Mouse updates
Download the latest "combo" OS updates (currently OS 10.6.1 for Snow Leopard and 10.5.8 for Leopard) and reapply them, along with the latest Magic Mouse update. Another way to test OS installation issues is to boot to the OS X installer and see whether the problem persists there. If there is a difference then it definitely indicates a software configuration problem.
Reset the computer's PRAM by holding down the options-command-P-R keys at boot-up, letting the system reset a couple of times with these keys held before releasing them and allowing the system to boot normally. In addition, reset the computer's SMC according to the instructions in this Apple knowledge base document.
Replace the batteries
I would not expect there to be low batteries in any relatively new Magic Mouse, but in the off-chance that yours was shipped with drained batteries, try replacing them.
Reset the connection and re-pair the device
Go to the Bluetooth and Mouse system preferences and remove the mouse from your system. Then turn the mouse off and back on so it goes into discovery mode and pair it back up with your Mac through the "Mouse" system preferences.
Hardware configuration issue?
If the mouse is more stable at closer distances, it indicates a problem with the computer's hardware setup, specifically the antenna configuration. This may be particularly true if you have troubles with other Bluetooth devices as well. A number of Apple's Mac Pro computers have shipped with improper antenna configurations for the airport and Bluetooth modules, and many people with these computers have been able to fix the problem by swapping some mislabeled antenna wires between the airport and Bluetooth cards.
This was noted several years ago, but the problem may still persist in new computers, or may have gone unaddressed in older computers that are only now being used for Bluetooth connectivity. The fix will require you to remove the door of your Mac Pro and detach/reattach the wires, which are just pressed on to their mountings so you should not have to unscrew any components of your system. For specific instructions on which wires to switch around, read the following documents:
Unfortunately, mislabeling of the wires may not get the problem properly addressed if you take the computer in for servicing at an Apple store; however, if you do not wish to mess with your computer's hardware, it may be worth discussing the problem and the potential wiring issue with a service representative to see if he or she is willing to test out wiring configurations. Sine the antenna connections are just press-fit, and Mac Pros are designed to be user-serviceable, you should be able to unplug them and swap them around without damaging the system. Just be sure to keep track of the original configuration so you can go back to it if necessary.