Hands-on with troubleshooting firewall corruption

The OS X firewall is a useful and important security feature, and we recommend people enable as many of its features as possible to help ward off the possibility of attacks, especially when connected to public networks. Despite its benefits, as with any software package the firewall is susceptible to corruption that can interfere with its functions, and result in odd behavior. One of these is the notorious self-assigned IP address problem that can plague various network ports in OS X.

Recently one of my computers (a PowerMac G5 Running OS X 10.5.8) suffered a power outage when in sleep mode, which did more than just turn off the system. Upon rebooting, the system seemed to work properly but it would not connect to the Internet. Both Airport and Ethernet connections claimed they had self-assigned IP addresses, and as a result I was not able to access e-mail, synchronize with MobileMe, or browse the Web. My other computers on the network were fine, but this particular one was not able to get an IP address configuration.

Rebooting the system did not help the issue, and starting up in Safe Mode also showed no promising results. Sometimes these issues can resolve themselves by just waiting, so I let the computer sit for about half and hour before attempting to restart and connect again; however, it was still a no-go. At this point I tried more in-depth cleaning routines, assuming the loss of power during sleep could have caused some cache corruption, and decided to run the full gamut of general maintenance routines that we commonly recommend for people in these situations.

After clearing the caches and resetting the PRAM, the system booted slightly slower than before upon initial boot (as expected), but then started showing a new behavior, which was to present me with requests to allow incoming connections for a number of system-related processes. Usually the system only asks to allow network access for user-installed applications, and sets up exceptions for system processes in the firewall when you enable those services in the system preferences; however, I was seeing requests for everything, including the built-in Web server, the Internet sharing process, and the Samba suite daemons, among others.

This indicated that in addition to the system being unable to communicate with the DHCP server in my router, there were problems with how the firewall was configured to handle networking processes in the system, and indicated a problem with the firewall. To see more details about the firewall's functions, I checked the firewall logs in the Console utility and found the following entries repeated many times:

Sep 7 06:36:04 Tophers-Desktop Firewall[63]: Deny configd data in from 172.29.62.12:67 uid = 0 proto=17
Sep 7 06:36:04 Tophers-Desktop Firewall[63]: Deny configd data in from 10.15.1.7:67 uid = 0 proto=17

These entries in the logs show the firewall is denying access to services running on port 67, which is one of the ports used for DHCP and BootP configuration connections (DHCP is how most network routers supply clients with an IP address configuration). Without these connections the computer cannot get a valid IP address, subnet mask, and gateway from the ISP or local router; however, OS X will still recognize a valid hardware connection through the network hardware ports and will assign a manual IP address with a fairly open subnet mask so you can still make local ad-hoc connections with other computers in the vicinity.

To test the issue further, I turned off the firewall by setting it to allow all incoming connections, and then went to the Network system preferences and renewed the DHCP lease to see if the system could get an IP configuration. This time the system did get a proper IP address and was able to connect to the Internet, so I next turned the firewall back on and renewed the DHCP lease one more time and found the system could not renew the lease.

It was clear that the firewall was not behaving properly, and was likely suffering from corruption in its settings. To fix the issue, I restarted the system in Safe Mode, and went to the system preferences and turned off the firewall. I then went to the /Macintosh HD/Library/Preferences/ folder and removed the firewall preferences file (called "com.apple.alf.plist") and then restarted the system normally. Upon logging in I checked my Network preferences to ensure I was able to get a proper Internet configuration, and there was a valid IP address, subnet mask, and router in the appropriate fields.

With the network now working, I went to the Security system preferences and turned on the firewall again. This time the firewall only prompted me to grant access for applications being launched or activated by my user account. Rebooting the system now maintained the proper IP configuration and no longer showed any odd behavior for the firewall.

Questions? Comments? Have a fix? Post them below or e-mail us!
Be sure to check us out on Twitter and the CNET Mac forums.