Distributed Tiger: Xgrid Comes of Ageby Drew McCormack
Waiter...there's a Distributed Computational Network in my Panthera Tigris!
Around a year ago, I waxed lyrical in these pages [1,2] over the then preview release of Xgrid, an Apple product for distributed computation. I made various predictions, some on the money, and others less so.
When Mac OS X 10.4 Tiger was released, Xgrid lost its preview status, and became a real boy. In fact, it's shipped in every copy of OS X Client and Server. Now, I would like to revisit Xgrid, bring you up-to-date on the changes, and show you how it could be useful in your application development, whether your interest is DNA or DVDs, Picasso or Poincaré.
In this first of two articles, I will show you how to set up a small Xgrid for testing purposes, submit simple jobs to the grid with the command line interface (CLI), and query their progress. The second article will be a Cocoa Tour de Force, involving new Tiger technologies like Automator and Core Image, in addition to Xgrid.
I'm going to take quite a bit for granted, and won't go into any detail related to the architecture of Xgrid. If you are not familiar with Xgrid at all, your best course of action would probably be to read the first of my original articles, before proceeding with this one. Apple has also provided a document on Xgrid Administration for Mac OS X Server, which gives a good overview of how Xgrid works.
Although Xgrid 1.0, which shipped with Tiger, is fundamentally the same as the preview releases that I wrote about in the earlier articles, there are some important differences. Probably the most significant is that Apple has now steered away from the relatively inflexible approach of submitting jobs with a GUI tool, and instead opened up the client-side Cocoa API for public consumption. This was one of my original predictions that turned out to be correct (he said smugly).
This move has a number of advantages; anyone who used the preview releases of Xgrid would soon have realized that supplying a GUI, and requiring developers to write plugins, was not the best approach for grid computing. The problem is that distributed computation can take so many different forms; there are as many applications as there are developers to implement them, and each application requires a completely different user interface.
What's more, many developers would rather integrate Xgrid functionality directly into their applications, as this provides for a better user experience than requiring users to install plugins and work with a separate Xgrid client application. In fact, the second of my original pieces on Xgrid demonstrated how you could do just that, even without a Cocoa API, by invoking the
xgrid command-line tool from within a Cocoa application.
Apple must have been listening to its developers, because Xgrid 1.0 does include a Cocoa API, which allows developers to directly integrate Xgrid into their applications without depending on command-line hacks. The original Xgrid client GUI has largely disappeared, but some of its plugins live on as examples in the
/Developer/Examples/Xgrid directory, which appears after you install the Xcode developer tools. The GridSample project is particularly useful because it allows users to run arbitrary serial and parallel (MPI) jobs.
Another benefit of opening up the Xgrid API to developers is that it paves the way for third parties to build more powerful clients, in addition to integrating Xgrid directly into their applications. Good examples of this are PyXG and GridStuffer. PyXG is a python interface to Xgrid developed by Brian Granger, and GridStuffer is a free Cocoa client developed by Charles Parnot at Stanford.
GridStuffer is similar to the GridSample application, but is considerably more advanced, offering many more options for job submission and monitoring. (Incidentally, Charles is grid master of one of the largest Xgrids in the world--maybe THE largest. He uses it for biochemical modeling, so if you have some spare CPU cycles, why not donate them to his research?)
Xgrid has not only evolved on the client side; the controller has also seen considerable change. My original prediction that Xgrid would facilitate peer-to-peer computation--where Macs would seek each other out via airport, and help each other perform expensive operations like iMovie renders--still seems some way off. Apple has instead opted for a more traditional model, with a central Mac OS X Server acting as controller. They have provided a basic controller on the client version of Tiger, but it is intended for testing purposes only, and doesn't have some of the security features (e.g., Kerberos Single Sign-On) of the controller supplied with the server version of the OS.
Configuring an Xgrid Agent
If you aren't concerned with security, setting up Xgrid is very straightforward. We will now go through the steps required to get a simple test grid up-and-running on your local machine. This grid will not use any form of authentication, so it should only be used on systems that are isolated from the big, bad world. Later, I will show you how you can add password authentication, which is more involved, but also offers more peace of mind.
First, we will configure the Xgrid agent. The agent is the guy that performs the calculations contained in a job. You configure the agent using the Sharing pane in System Preferences. If you open the pane, and select the Services tab, you should see Xgrid in the table view on the left. Select it, and click the Configure... button. Configure the agent to connect to the first available controller, always accept tasks, and not use any authentication. Figure 1 shows how it should look.
Configuring the Xgrid agent in System Preferences.
Not all Macs run Mac OS X 10.4, so you will be happy to hear that there is an agent for 10.3 (Panther) clients available from Apple's website here.