Troubleshooting OS X applications: a case study
Troubleshooting OS X applications: a case study
Background
We have been using the same email database for several years. It began life with Outlook Express 5, migrated to Entourage 2001 and finally to Entourage X. The file has been rebuilt several times, and once we have had to restore from a backup because a crash rendered the file unreadable. (Had we not been able to recover the data file this time, we had a backup that was less than 24 hours old. The damage would have been minimal.) At present the file contains over 67,000 messages.
We were in the process of archiving a large number of old email messages into FileMaker Pro with an export script we wrote several years ago. It is interesting to note that under Mac OS 9, exporting would tend to halt because of a memory error after about 500 messages, however, under Mac OS X, the same script moved over 14,000 messages in a single batch without complaint.
How it started
The trouble began when we tried to delete the archived messages. The 14,000 messages were deleted in a single batch, and because we have Entourage set up to empty the deleted messages folder on quit, we quit the program. At some point while removing the messages from the deleted items folder, Entourage crashed, and subsequent attempts to open it resulted in the program unexpectedly quitting almost immediately after launch. In hindsight, it would have been far better to increase the number of batches each with a smaller set of messages, but we were so impressed by the performance of the export script that we didn't give it a thought beforehand.
Attempted recovery
Assuming the database was damaged, initially we attempted to rebuild it by holding down the option key while launching Entourage. Both the typical and advanced rebuilds, each run several times, all resulted in a message that, after substantial processing, said in effect the database could not be rebuilt. Each time we launched Entourage, the program would quit as soon as the splash screen appeared. We also stumbled on the fact that Word would no longer launch, either. That discovery lead us to the conclusion that something common to both programs was damaged, in addition to the corrupted database. Our initial suspect was Office Notifications, however, we eventually found that removing all traces of the 'bad' Microsoft User Data folder from the Documents folder still caused both Entourage, Notifications, and Word to crash on launch. Another discovery made at this stage was that moving Microsoft User Data folder out of its standard location in the Documents folder was not enough to disable it. Although a new User data folder was created in Documents, Office appeared to still be opening at least some of the files in the previous data folder. It seems the only way to be certain which data folder Office is using to have only one on the drive and in the Documents folder.
Curiouser and Curiouser
Forget about restoring the data files, now we just wanted to fix whatever was wrong with Office. This lead us to a conundrum: it seemed we had isolated the data file corruption, and one or more of the Office files had become damaged as well, but which one, and how do we fix it? We tried manually removing the Office preference files we could identify in Library/Preferences, without success. Reinstalling Office via the installer did nothing either: it reported that Office was fine and did not need to be installed. What finally worked was to delete the entire Office X folder in Applications, log out and back in, drag a new folder from the CD, and launch one of the programs. That got Office X running again. And since the unexpected quit error occurred with both the damaged database file and a fresh copy, we once again attempted an advanced rebuild. This time, the file was restored.
The bottom line
The ultimate cause of this problem was that we probably pushed Entourage harder than was prudent, but the result did teach us a few lessons. Troubleshooting a major application under Mac OS X is both easier and more difficult than under previous versions of the Mac OS. Because we didn't need to reboot countless times, we found it much faster to isolate what turned out to be essentially two separate corrupted files, the database and at least one file inside the Office folder. Without rebooting regularly, we did however need to be careful to log out and back in on occasion to be sure that all the files were closed properly before moving anything around. Moreover, we were dependent on our previous knowledge of where Office stored its data, and we were lucky that the drag and drop reinstall weeded out whatever went wrong aside from the corrupted database.
What have we learned that would improve OS X troubleshooting in general? At present, there is an option to replace the OS itself with a fresh install, but applications are less consistent. (Office X is quite good in this respect with its drag and drop install method, but we would have liked to see a bit more documentation for support files.) At minimum, we would like to see a provision to reinstall any application so that every file is replaced with a fresh copy, along with clear documentation about the location of every application-created support and preference file. A remove option would be even more helpful, especially if it removed every support file that did not contain user-created data. [Update: There is a Remove Office application in the Value Pak folder that does most of this.] Care needs to be taken also with respect to files left open after a crash, so that the files can be removed or repaired. Another improvement would be to expand the use of application packages as much as possible, so that reinstalling an application is as simple as dragging a fresh copy of a single icon from the CD to the hard drive. Given that most major applications are requiring hundreds if not thousands of files, extensive documentation on the locations and contents of user-created documents, application files, and application support files is crucial. [RD]