That's the bitter truth. It's only one RF channel and you share the 11 or 54 megabit bandwidth.
There's also some other issue with 32 clients. To support 50, you need a minimum of 2 WAPs on different RF channels.
How many clients can an AP ( Cisco 1200 b/g ) handle without any noticable performance degradation in network speed . Is there any high end AP which can handle around 50 clients.