It's not a "ridiculous" limit, it's one that's been in place since the days of the Intel 386 processor back in 1986. Put simply, 4GB is the maximum amount of RAM that you can fit into a 32-bit variable, and the reason you see only about 3.5GB in XP is because part of those 32-bits are reserved for system functions like protected mode execution. This is what keeps Application A from clobbering the memory address of Application B, resulting in the all too common General Protection Fault (GPF) errors from the 16-bit Windows days.
This has been a known limitation of ALL 32-bit processors and operating systems for over 20 years, so why it comes as this giant surprise to people is a bit of a mystery to me.
And yes, there is Physical Address Extension (PAE) features on most Intel and AMD CPUs, but there is simply no good way for Microsoft to support this in any of their 32-bit operating systems. If they added support for this feature, they'd have to field calls from people whose CPUs do not support this feature and are upset because it says on the box that it can support more than 4GB of RAM. Much easier to deal with people like you, and simply point you to Vista x64 as a solution.
Since you already know about Vista x64, but are choosing to ignore the solution, then you're screwed, it's as simple as that. You could run Linux, which can and does support PAE if you compile it into your own kernel, but if you're not willing to upgrade to Vista x64, I suspect Linux is similarly out of the question.
For what it's worth, I've been using Vista x64 since around mid-July, and it's not half bad. For most people, I'd say there's little point in upgrading from XP, but in your case... I guess it depends on how badly you want to use that extra 2.5GB of RAM. Microsoft will NOT be supporting more than 4GB of RAM on any of its 32-bit operating systems, and there is no way to "crack/fix/patch" the system into doing so. You want to use that full 6GB of RAM with Windows, you'll have to get a 64-bit version, end of story. No point whining about a 20 year old limitation. If you want to get started whining about the 48-bit memory addressing limit on the x86-64 (or EMT64 if you have an Intel CPU) instruction set, go right ahead. At current rates, it'll probably be 5-10 years before we start bumping into that limit, so you've got plenty of time to make yourself look like some kind of tech prophet to the unwashed masses.