If you want to remove any component of Office, you need to click that "Remove" for Office 2010. Windows should then ask you for your Office CD, and then you remove the 64 bit version of Outlook.
You could then install the 32 bit version of Outlook whilst you are there, (although I didn't know the Office CD had both 32 bit and 64 bit versions), but I would be cautious. I would remove the CD once the uninstall has completed, then restart the computer, then insert the CD and add the 32 bit version of Outlook at that time.
Mark
I recently upgraded to Office 2010 and as my PC is 64bit, I went ahead and selected the 64bit version of Outlook to install.
I've learnt that this has a number of compatibility issues, particularly with syncing calendars and would have been better off installing the 32bit version.
Unfortunately, I wasn't able to uninstall Outlook 64bit through the Remove Programs section in the Control Panel. I've reopened the installer, which offers the following options:
"Add or remove features"
"Repair"
"Remove"
"Add a product key"
I tried clicking "Remove", but this gives me a dialogue box asking if I want to remove MS Office Professional Plus 2010 (instead of just Outlook).
So I clicked "Add or remove features" and selected "Not Available" for Outlook. It seems to uninstall and disappears from my programs, but when I go to reinstall it, it doesn't offer the option of 32bit - it goes straight to installing 64bit again.
Can anyone suggest how to let me do a 32 bit reinstall?! Would love to get this working, working!
Cheers
Gus

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic