Since the introduction of Mac OS X, its Objective-C-based Cocoa frameworks have allowed developers to produce some amazing software at an incredible pace. All of the resulting innovation in the marketplace has made it pretty easy for us to stay captivated on the desktop and forget all about the engineering marvel that makes all of this possible. Let's take a trip down into the core of OS X--Darwin--to learn more about the foundation that gives us one of the best user experiences in computing.
Mac OS X development started shortly after Apple's acquisition of NeXT all the way back in 1996. After the buyout, development on what would evolve into Mac OS X was code-named Rhapsody until Mac OS X was officially released in 2001.
Sometime during Rhapsody's development, its code was forked and open sourced into what we know as Darwin. By April 2000, Apple's team had already completed their version 1.0 build. Although they'd been releasing the Darwin source prior to that time, the version 1.0 build was especially significant because it was based on a new hybrid core that we now know as the XNU kernel. Like the Free Software Foundation's recursive acronym GNU, XNU stands for "XNU is Not Unix."
Now that you know a bit about Darwin's past, let's take a moment to look at where it fits into the overall OS X architecture before diving off into the details. Generally speaking, most of Darwin fits into what's called the Core OS, which sits directly on top of the hardware and comprises the kernel, device drivers, and other low-level resources of the OS. Other layers, including core services, application services, graphics, and multimedia, sit between the Core OS and the user experience that is defined by Aqua: the standard that defines the overall appearance and behavior of Mac OS X apps. For some good reading on all of the ins and outs of how all of the OS X technology fits together, check out the Mac OS X Technology Overview. It's a good way to spend a rainy day.
Two different views of the OS X System Architecture. Most of the technologies in the "Core OS" are part of Darwin.
A good starting point for discussing any OS is the kernel: the core of the OS that provides access to and scheduling for the underlying hardware that's available. To fully appreciate Darwin, you'll want to know some of the details associated with its kernel, XNU, which is an interesting blend of the Mach 3.0 microkernel that's been modified to include portions of FreeBSD inside the kernel space for performance reasons. Mach's foundations go back to Carnegie Mellon University out of Pittsburgh while FreeBSD is a descendant of BSD (Berkeley Software Distribution), which hails from--you guessed it--the University of California at Berkeley.
XNU is the result of an effort to combine the strengths of two different approaches to kernel design: the monolithic approach and the microkernel approach. While the jargon might already sound complicated, the primary difference between the two is pretty simple and basically boils down to what programs run inside the kernel itself and what programs run in userland. Whereas monolithic kernels typically are defined by a fairly complete set of rich abstractions that access the hardware all within the same address space, microkernels tend to provide a fairly minimal set of services within the kernel and use daemon (server) processes running in userland for the rest of the core services.
Conceptual views of monolithic kernel (left) and microkernel (right) design
There are frequently debated trade-offs between the two approaches, but just about all of them eventually come back to efficiency as the pivotal point of discussion. The tightly coupled nature of a monolithic kernel allows it to make very efficient use of the underlying hardware, which is a very desirable thing. The drawbacks of monolithic kernels, however, are that they're very difficult to write and a single mistake in the kernel's code can crash the entire system.
Microkernels, on the other hand, run a lot more of the core processes in userland. This approach simplifies the actual kernel design and can provide a great deal of stability in the system. Unfortunately, these benefits come at the cost of the microkernel having to pass a lot of information in and out of the kernel space through a process known as a context switch. Context switches introduce considerable overhead and therefore result in a performance penalty.
You're probably beginning to see the underpinnings of the debate that unravels when you start to make a case for one kernel design over the other. Perhaps the most famous of all these debates was between Linus Torvalds and Andrew Tanenbaum. Torvalds makes a case for the monolithic kernel, while Tanenbaum defends the microkernel approach. You can read the original messages that were part of a world famous flame war back in 1992 here. (It's fortunate for us that the decision for Darwin's kernel has already been made.)
We've been harping on XNU for a while now, but there's still a lot more to be said about Darwin besides its kernel. Some of the most notable features worth mentioning include its facilities provided through I/O Kit, its Virtual File System (VFS), and its networking support.
Darwin's I/O Kit (Input/Output Kit) is a Mach-based device driver framework. It's modular, extensible, and offers a true plug-and-play solution for dynamic device management. Unlike Cocoa, which is written in Objective-C, the I/O Kit is written in a restricted subset of C++ that's based on the Embedded C++ specification. Much of the justification behind the choice of C++ as a language for I/O Kit was based upon C++'s mature compiler and long-standing support for systems programming.
Fitting into the Mach-based kernel methodology, device drivers derived from I/O Kit are special types of kernel extensions that provide the information necessary to handle a device or entire family of devices. Unless you're writing device drivers or interfacing to hardware at its lowest levels, you probably won't need to work with I/O Kit. But just in case you do, I/O Kit Fundamentals is a great starting point.
Darwin's file system and networking support comes from its rich BSD heritage. Its VFS design enables features like long filenames, URL-based volume mounting, and journaling to take place. Darwin's file system can handle just about anything you throw at it: Mac OS Extended/Standard, UFS, UDF, NTFS, and FAT. As the web becomes more and more pervasive, complementary networking support for file system support is a must have, and don't worry--you won't be disappointed.
If you've ever tried to set up networking on a Windows box, you know that it can often be a nightmare. At one point, you might have been changing all sorts of settings on multiple tabs that opened new windows, which also had tabs that had multiple properties with a poor help system. But with Darwin, there's none of that. Darwin's BSD-based extensible network protocol stack gives rise to OS X features such as Bonjour, a zero-configuration networking implementation that eliminates just about all of the hassles that are normally associated with networking. It really can't get much simpler than that. Although features like these are easy to take for granted, they really are gems in Darwin's overall design.
Apple releases a new version of Darwin on their open source site for each major update to OS X, and it's not long after Apple's release that the folks at OpenDarwin.org provide a corresponding build of OpenDarwin. OpenDarwin is essentially Apple's latest release with modifications to give it some extra features that Apple doesn't provide. These additional features are usually things like X11 GUI support, device drivers, utilities, and support for legacy hardware--basically features that don't directly support Mac OS X.
Although the releases of Darwin under the versions less than 2.0 of the APSL (Apple Public Source License) were criticized by the FSF, the APSL as of version 2.0 does qualify as a free software license, and this is the license that most of Darwin is released under. The FSF, however, considers the APSL 2.0 to still have two primary flaws: it allows linking with other files that can be entirely proprietary, and it is generally incompatible with the GPL. I don't think we'll be seeing any trade secrets like QuickTime, Spotlight, or the Cocoa frameworks being open sourced anytime (ever), so don't hold your breath on this one.
You can download Apple's version of Darwin or OpenDarwin if you want to tinker around and get into the guts of Mac OS X. If you're interested in trying out yet another possibility that comes with some additional bells, whistles, and software mods, take a look at GNU-Darwin--a project that aims to be the "most free" version of Darwin.
When Steve Jobs announced the Intel switch, a lot of people freaked out and wondered how on earth an entire OS could be turned around to run on a different architecture in such a short amount of time. Little did they know that Darwin had been leading the way by compiling for x86 alongside of Mac OS X since the conception of OS X--a practice that no doubt provides a significant boost to the overall effort. In fact, you can read about Darwin's 1.0 build right here in an old Advogato diary entry. I guess if you couldn't be there, then this is the next best thing. In the entry you'll even see a reference to fat binaries (universal binaries), a technology we've been hearing a lot about lately that also dates back to the days of NeXT.
For a good grin, check out I've Personally Heard Steve Say They Would Never Do That--a weblog entry that comments on an old Wired news article from 2000. It has some interesting quotes regarding the Intel switch in light of what we now know today.
Now you'll never have to wonder what that friendly little message you see in Terminal ("Welcome to Darwin!") ever means again. Hopefully this background has provided you with a better understanding of how Mac OS X works, where it comes from, and how it powers OS X. Now get out there, download the Darwin source, and take a look at how OS X works under the hood.
Matthew Russell is a computer scientist from middle Tennessee; and serves Digital Reasoning Systems as the Director of Advanced Technology. Hacking and writing are two activities essential to his renaissance man regimen.
Return to the Mac DevCenter
Copyright © 2009 O'Reilly Media, Inc.