Published on MacDevCenter (
 See this if you're having trouble printing code examples

Mac Security: Identifying Changes to the File System

by Peter Hickman

Editor's note: Before you get knee-deep into this very interesting article that discusses, among other things, a hacker's toolbox know as rootkit, I want to point out two things. First, the way "hacker" is referred to in this article is in the nefarious sense, as in "black hat," not in the playful spirit of the phrase that we usually think of. Second, keep in mind this passage from the author: "There are few rootkits tailored for BSD (the codebase underlying OS X) and none specifically for the Macintosh. Talk of Macintosh rootkits is less credible than Elvis sightings, but as we are at heart running a Unix system, a generic Unix rootkit could be deployed on your Macintosh. Just remember that security is all about planning for the worst case; it pays to be paranoid. However, the sky is not falling; not today, at least." That being said, enjoy this piece. It's fascinating.

When you use a Macintosh, or indeed any Unix-based system, it's comforting to know that your computer is more secure than Windows. It is, however, a mistake to equate "more secure" with "invulnerable." Keep in mind that the hacker who does break into your system has more skills than the average script kiddy. And when this happens, what should you be looking for? Let's assume that a malicious hacker wants one of two things:

  1. To steal valuable data from your computer, such as credit card numbers or passwords.
  2. Access to your system so that they can cover their tracks as they hack into other peoples computers.

The Anatomy of a Break-In

Although I said that our hacker will be more skilled than the average script kiddy, they often have something in common: pre-packaged software. Once they have broken into your system they will download a bundle of tools called a rootkit. The purpose of a rootkit is to secure access to your machine and cover their tracks.

When working on your computer, they need to store files and be sure that you will not stumble across them and get suspicious. To this end, the rootkit will provide replacements for several basic Unix tools such as ls and find. These modified programs function identically to their legitimate counterparts, with the exception of omitting reporting the existence of the directories and files that the hacker has created.

Related Reading

Mac OS X Tiger for Unix Geeks
By Brian Jepson, Ernest E. Rothman

The hacker will also want to run some programs of their own and be sure that you'll not notice them. Again, they'll provide replacements for any program that can be used to monitor running processes, such as ps or top. Here is a list of compromised programs found in various rootkits:

adduser amd basename cat chattr checkproc chkconfig chmod chown chroot cron csh date depmod df dmesg du echo ed egrep env fgrep file find grep groups head id ifconfig ifdown ifstatus ifup inetd init insmod ip kill killall kldload kldstat kldunload ksyms kudzu last lastlog less locate logger login ls lsattr lsmod md5 md5sum modinfo modload modprobe modstat modunload more mount netstat newsyslog nologin passwd ps pstree rmmod runlevel sh sha1 sha1sum size slocate sockstat sort stat strace strings su sulogin sysctl syslogd systat tcpd test top touch uname useradd usermod users vipw vmstat w watch wc wget whatis whereis which who whoami xinetd

The list is mostly for Linux systems, but you will notice quite a lot of familiar commands in here. Of the 104 files listed, 63 of them exist on the Macintosh. Once your computer is compromised, and you cannot trust your own keyboard, what do you do?

Looking for Clues

Reining in our paranoia for a moment, let's make a point clear here: There are few rootkits tailored for BSD (the codebase underlying OS X), and none specifically for the Macintosh. Talk of Macintosh rootkits is less credible than Elvis sightings, but as we are at heart running a Unix system, a generic Unix rootkit could be deployed on your Macintosh. Just remember that security is all about planning for the worst case; it pays to be paranoid. However the sky is not falling; not today, at least.

If you think that your machine had been compromised there are tools you can use to scan your system for known rootkits such as Rootkit Hunter and chkrootkit. Rootkit Hunter runs on OS X and chkrootkit does work on BSD-based systems, but not specifically OS X. These tools look for evidence of rootkits by a variety of means, but are in part based around the idea of checking the size, permissions, and checksum of a known target file or program, such as ls, with a database of known good values.

Say Hello to mtree

mtree is a tool that comes from the Macintosh's BSD heritage. It is designed to do quite a few things, but its original purpose was to map a directory tree so that you could later check the permissions were still valid or recreate them on another machine. First, it allows you to create a specification file of a portion of your directory that records the following:

From this specification file, you can:

  1. Compare the specification to the current file system and report the differences.
  2. Apply the permissions and ownerships from the specification to the file system to restore the correct values.
  3. Recreate the directory structure elsewhere with all of its ownerships and permissions.

There's much more that mtree can do, but for our purposes, option one looks like an interesting avenue, especially as mtree did not appear in the preceding list of common rootkit tools. We're going to create a couple of programs that can report changes to the files on a computer. My root volume has 455,123 files on it, so any tool that can help me check through them all to find the changes is a big help. Let's write some code:


# The volume that we want to check

# The file of directories to exclude

# Where we write the report to

/usr/sbin/mtree -Pcx -k flags,gid,mode,uid,size,cksum \
                -X $EXCLUDE -p $TOCHECK > $REPORT

The first variable is TOCHECK and is the volume that we want to check, in my case, the root volume of my system. This is followed by EXCLUDE, which is the name of a file containing a list of files and directories that I do not want included in the check (we will come to this list later), and finally, REPORT is where we will write our report. Once run (and it will need to be run as root), it will create a specification file that documents all of the files and directories and their associated data. On my Macintosh, it takes around 19 minutes to process 14GB of data. Let's call this mtree_build.

A quick run through the parameters (all of which are available from the man page): the P stops mtree from following symbolic links but treats them as files. The c tells mtree write the specification that it creates to standard out, where we will capture it into REPORT. The x stops mtree from crossing mount points; this means that if you have an external hard drive attached, then mtree will not include it in the specification it creates. The -k tells mtree the sort of information that we want to record about the files and directories that it encounters. Specifically:

There are other flags, but these will allow us to monitor if a file or directory's ownership, permissions, or contents have changed, and that is enough for our purposes. Now we have a specification; we need some means of checking it against the current file system to see what has changed. Another piece of code is called for:


# The volume that we want to check

# The file of directories to exclude

# the old specification

# Where to write the report to

/usr/sbin/mtree -Px -k flags,gid,mode,uid,size,cksum \
                -X $EXCLUDE -p $TOCHECK \
                -f $SPECIFICATION > $REPORT

Strangely similar to the first piece of code with only one change, we are no longer creating the specification (dropping the c flag), but reading it from the file we created in the first script (by adding the -f flag). mtree then compares the current file system against the SPECIFICATION and when it finds any difference, files, or directories added or removed, permission and ownership changes, or contents altered, it will write to REPORT. Let's call this mtree_check and, after a suitable interval, run it, again as root, and look at the report.

Various Examples

private/var/log/mail.log changed
    size expected 25917 found 742
    cksum expected 3652242742 found 1276740875

Log files are altered and rotated on a daily basis, and as such, any change to a log file is not necessarily a red flag. We need not be too worried about this one. But the next one is more of a problem:

private/var/log/system.log changed
    permissions expected 0640 found 0662
    size expected 49066 found 18671
    cksum expected 3572187358 found 1678708264

Here the system log file, which is usually only writable by root, has been made writable by everyone. There's no good reason for this, and no normal program would have done this. If you didn't do this, then you should be suspicious of the change. Change the permissions back and examine the file; let us assume that someone has set write permission on the file so that they could alter its contents to cover up their break-in. Look for gaps in the timestamps. My system runs crons every five minutes, so I would be looking for any missing cron entries that would give me a clue as to when the event took place. Alas, it does not tell me what took place, only that something did and maybe when. Here's another:

usr/bin/passwd changed
    cksum expected 426431686 found 2543813206

The only change reported is the cksum. From this, we know that the contents of the file have been changed (although we do not know what the actual change was). This is a basic system utility, and you don't just recompile it on the fly; indeed, you won't find the source for this on your Macintosh unless you went to the Apple site and got it specifically. Now would a good time to start panicking, especially if you found several files had changed. Fortunately for me, this was just the result of installing an updated version of Mac OS X, and the panic was short-lived.

This is a problem with the report--lots of files get changed in the normal usage of a system. I left mine alone for 24 hours, and 38 files turned up in the report. All logs get added to and moved as part of the daily periodic tasks, and applications such as SoftwareUpdate update their plist files. When you actually use your computer for even the most mundane tasks, such as email and browsing, the list gets large very quickly. There are three things you can do:

  1. Diligently check each line of the report.
  2. Write a program to filter the report and highlight exceptional activity.
  3. Exclude irrelevant files from the specification.

I go with a combination of 2 and 3. Changes to executable files worry me more than those to other files, and changes to permissions and ownership more than size changes. So I have a program rank the changes according to severity, and I can then investigate the most severe changes. I will leave the writing of such a program as an exercise.

Excluding Files from the Specification

To exclude files or directories from the specification, mtree takes an -X parameter, followed by the name of the file that lists the files and directories to exclude. Whatever you do, make sure that you do not exclude your exclude list from the specification. Mine looks like this:


Being a developer, my home directory has frequent changes that put too many false positives into the report. So I have chosen to ignore the home directories. The Fonts directory has a cache that keeps changing, and the Logs directory always changes. The lines in the exclude file are supposed to be regex patterns, and should allow a great deal of control over what files are to be excluded, except that it doesn't seem to work. Note also that as we created our specification with the -P option, to make it treat symbolic links as files, we are not able to add /tmp to our exclude file. We would need to add /private/tmp, as this is the actual directory that /tmp is a link to.

What to Do if You Have Been Rooted

If you have been rooted, then you are truly buggered, as we say in England. All that a tool can do is confirm what you've only suspected. There's no program that will clean your system beyond identifying the programs that the rootkit replaced. But a rootkit is only the starting point, and no tool can know what the hacker might have done to your system.

Have they changed your Apache configuration? Have they placed a script in your cgi directory that's waiting to be run to re-infect you? Have they placed a key of their own in your SSH keys file to allow themselves to log in? Have they set up a cron job to see if they have been removed and automatically re-infect you?

There are just too many unknowns, and little you can do to regain control of your system unless you know each and every file that has been compromised. Unfortunately, the hacker is the only one who knows, and he's not telling. This is what you do:

  1. Turn your computer off.
  2. Disconnect it from the internet.
  3. Boot from your installation CD and select an install that preserves the User directories.
  4. Once the install is complete, copy any user data you want to keep to an external drive. Do not copy any executable files.
  5. Reformat your hard disk and do a completely virgin install.
  6. Connect to the internet and run SoftwareUpdate until there are no more updates.
  7. Reload the user data you saved.
  8. Reinstall your applications from their original CDs or download them anew.
  9. Make a backup.

Yes, you have just wiped your hard disk, and no, there isn't another way. If the file is executable, be it a shell script, Unix command-line tool or Macintosh application, you can no longer trust it unless you download it from a reliable source or install from CD. Even your backups are of limited use. Unless you know the exact day that your system was hacked, you do not know which backups are clean and which are compromised. So get the user data from the backups, but not the applications.

The problem now is that your system is in the same state that it was when the hacker broke in. What's to stop them from breaking in again? Nothing, if you have duplicated your original setup. So let's do a quick review:

Choose a better password, make it longer, with mixed case, numbers and special characters. and tell no one. Turn sharing off and the firewall on. For now, at least, turn off all of the services you can. Once you have calmed down and the hacker has given up and moved on to someone else's machine, you can turn them on if you need to.

If you have a backup that you absolutely know was not compromised, it is possible to replace each file with a clean copy. You can then hunt down any suspicious files that are on your hard disk and remove them. But let's be realistic for a moment; your Macintosh has hundreds of thousands of files. Do you know each one by name, and what they should look like? Reformat your disk, and you will sleep much better.

Using our Scripts

I have a daily cron entry that runs just after my daily backup to create a report and a new specification file. This is run from the root crontab:

0 2 * * * /usr/local/bin/mtree_check; /usr/local/bin/mtree_build

I also keep a copy of the scripts, mtree itself, and a known specification file on a USB flash drive in case I feel the need to be absolutely sure everything is OK.

Other Tools

Our exploration of mtree has provided us with a useful tool to detect a rootkit on our system, but there are others available. Brian Hill has a tool called CheckMate that checksums critical system files and can alert you when they change, all nicely packaged up as a control panel. If you do not feel confident in your scripting skills, but want to be prepared, this will help.

Of the tools mentioned earlier, only Rootkit Hunter can be downloaded, installed, and run without having to dive into the code. It also doesn't spit out error messages for no good reason. It monitors critical system files and does other checks for signs of rootkits. Although not as functional on the Macintosh as it is on Linux, it is in active development and what it does it does well.

Final Thoughts

All of this effort, and there is no rootkit for the Macintosh. It is only a matter of time before one of the BSD rootkits becomes adapted for the Mac, and hackers start to target the Macintosh more aggressively. The Macintosh is a Unix box, and Unix boxes have rootkits. The conclusion is hard to avoid. One day there will be a Macintosh rootkit, and when that day comes, we will be ready. Besides, we now have a nice tool that we can run before and after installing a new application to see what files it installs and where, which will help with cleaning up after when we uninstall it. A bonus for our paranoia.

Peter Hickman is currently working as a programmer for Semantico, which specializes in online reference works and Access Control Systems. When not programming or reading about programming he can be found sleeping.

Return to the Mac DevCenter

Copyright © 2009 O'Reilly Media, Inc.