SSH service connecting but not authenticating in OS X

If you are even slightly familiar with the OS X terminal, then SSH (remote log-in) is a great service to have enabled on a system, especially for troubleshooting purposes.

In instances where your display is frozen or blank, or if your system is not accepting input from keyboards, being able to remotely log in and at least run a shutdown command to avoid a hard reset is a beneficial option to have.

Apple makes setting up SSH easy and convenient by just enabling the service in the Sharing system preferences. But even with this ease, sometimes establishing a basic connection with SSH may fail.

There are a number of reasons why an SSH connection may not work properly, especially if you are connecting between two different networks where hardware firewalls and routing might need to be contended with; however, there also might be instances where failures can happen when you have a basic setup of two systems on the same network.

Recently I ran into such a situation, where my Mac running OS X Lion froze when waking from sleep, showing the log-in prompt but the cursor was locked in place. While the log-in screen was frozen and not accepting input, the system was issuing error sounds in response to key presses, indicating the OS was functioning fine under the frozen screen.

In this situation, though a hard reset would be one course of action, the computer had SSH enabled so using the Terminal command "shutdown -r now" would have provided a safer and softer restart; however, the connection failed every time claiming an authentication error, even when supplied with proper credentials for an administrative user. In such a situation, people might be a bit perplexed as to why the system could establish a proper connection but not authenticate properly.

It turns out that while the system was configured to have SSH enabled, a simple oversight prevented the service from being used. When you enable SSH, by default it will allow all local user accounts to log in, but you can also select the option to only allow specified users, which is a more secure option for running SSH. Unfortunately, if you select this option you will also need to specify a user account, so if for some reason you have not then while remote systems can establish an SSH connection, the system will not allow any accounts to authenticate.

This was the situation with my Mac, which forced me to perform a hard reset instead of using the Terminal to overcome the problem. Going to the remote log-in service settings showed that the system did not have any accounts specified for SSH access. Therefore, if you have enabled sharing services, especially for troubleshooting purposes, you might want to double-check that you have them set up for proper access.

The main options here are the Remote Login and Remote Management (or Screen Sharing) services, all of which have an option to limit access to a select group of users. For these services to work, you will need to either allow access for all users, or ensure a local user (preferably an admin account) is listed in the access list.

In addition to the settings for the services, the other system configuration that might prevent connectivity is if you have the firewall enabled and set to block all incoming connections. This option is the most secure, but will prevent your system from receiving any incoming connection requests for the various sharing services you might have enabled. To check whether or not this is enabled, go to the Security system preferences and in the Firewall settings locate and uncheck the option to "Block all incoming connections."

Questions? Comments? Have a fix? Post them below or e-mail us!
Be sure to check us out on Twitter and the CNET Mac forums.