The default file-sharing protocol for OS X is Apple's "Apple Filing Protocol" (AFP) service, which is the primary service enabled when you select File Sharing in the system preferences. Enabling this will allow Macs on the same network to discover and connect to shared folders on the system, but there may be times when an error will occur that prevents users from logging on. When this happens, the client systems attempting to connect will get an message that states "this file server will not allow any additional users to log on."
This error happens because of an inherent limitation in the file-sharing services for the client version of OS X (the default version that ships with most Macs). In the file-sharing service in the client version of OS X, Apple has imposed a 10-user limit for AFP connections that is hard-coded into the service so it cannot be changed by merely adjusting a hidden configuration setting.
While the pre-set limit will result in this error if your system receives more than 10 active connections, it may also occur with fewer than that, since the system by default will not drop idle connections.
This setup was initially intended to prevent the need for regularly logging back in when accessing a connection; however, with the ability to store log-in credentials in the OS X keychain, the need to preserve idle connections is not as important. Therefore, you can help prevent these errors by enabling the disconnection of idle connections. To do this, open the Terminal utility on your system and run the following command:
defaults write /Library/Preferences/com.apple.AppleFileServer idleDisconnectOnOff true
After this command is run, restart your Mac (or disable and re-enable File Sharing in the system preferences) and see if the system now better manages file-sharing connections.
In addition to this command, you can adjust the length of time your system will wait to determine when an idle connection is dropped. The default is 10 minutes, but you can adjust this accordingly by using the following command (change the '10' value to the number of desired minutes):
defaults write /Library/Preferences/com.apple.AppleFileServer idleDisconnectTime 10
If you regularly need to serve files to more than 10 active connections on your network and are finding that, despite tailoring the idle disconnect feature, the client systems are receiving connection errors, then you will have to upgrade to OS X server. This is a $20 purchase from the App Store that is an add-on to your current OS X installation (no need to reinstall in order to upgrade). Among other options, this will give you an AFP service that supports unlimited connections and allow you to better serve your network.
The upgrade process should open the number of allowed connections; however, if you upgrade and continue to run into connection limits, then be sure to access the OS X Server configuration utility and adjust the number of allowed connections to a higher level, as the 10-client limit from before upgrading may still be preserved. You can adjust these settings in the Terminal by running the following commands, followed by disabling and re-enabling File Sharing:
defaults write /Library/Preferences/com.apple.AppleFileServer maxConnections 10
defaults write /Library/Preferences/com.apple.AppleFileServer maxGuests 10
Note that these last two commands also affect the Client version OS X as well as the server. While the client is overall limited to 10 connections, it still reads these two settings to determine if a quantity that's less than this value is set to be the maximum number of connections. If for example, an error has resulted in these values being set to 0, then even with file-sharing enabled the system will not allow any connections. In this case, try running these commands to ensure the maximum allowed connections are set to the 10-user limit allowed for OS X client.