MacDevCenter    
 Published on MacDevCenter (http://www.macdevcenter.com/)
 See this if you're having trouble printing code examples


A Closer Look at Spotlight

by Matthew Russell
05/24/2005

Apple aggressively revs up its operating system with each new release, and Tiger carries on the tradition. One of Tiger's coolest new features is a powerful search engine called Spotlight. In this installment, we'll look at how Spotlight works and how it can revolutionize your workflow in a variety of ways.

Do What I Meant, Not What I Typed

To really understand the power of Spotlight, let's review the basic problem search engines face today--being able to return information that you "sort of asked for." For example, let's suppose you searched the Web for "michael jackson" during April 2005. Given the legal proceedings at that time (the real-world context), you'd probably have been expecting to get back some hits about the court hearing, and would want the most popular stories to be your top hits--even though the bland search term "michael jackson" doesn't specify any such context on its own.

What if you'd searched for "bill cosby, comedy, god"? If you're familiar with Cosby's hilarious standup acts involving religion, you would have been expecting back some media clips from Bill Cosby Is a Very Funny Fellow Right!. Using just those search terms, however, the expected results were not very obvious--not to mention that media clips can be difficult to categorize.

How should a search engine categorize a collection of images and/or sound in a binary encoding? Unless the clips have really descriptive filenames or internal comments of some sort, search engines have a difficult time classifying these sorts of things.

It should be fairly clear that accurately classifying gobs of files in arbitrary formats is an incredibly difficult task. It's like one of those standardized tests where many of the correct answers turn out to be "not enough information to solve the problem." Don't you hate that answer? Fortunately, Spotlight is a search engine with the luxury of operating in a fairly specific and controlled environment--your machine and any volumes you mount--so let's look at how it tackles the problem.

How Spotlight Works

Let's consider the task of classifying an arbitrary file, the basic problem Spotlight encounters. If we were doing this manually, many of us would start by looking at the file's name to see if it is self-descriptive or contains any keywords. We'd take note of the file extension (if it had one) and possibly try to open it up in an appropriate application to view the file's contents. Assuming we used the appropriate application, we'd eventually succeed in opening the file and be able to assign it some classification.

Related Reading

Learning Unix for Mac OS X Tiger
By Dave Taylor

Now, consider the same problem, but from Spotlight's point of view. Ideally, Spotlight should accurately classify a variety of arbitrary file formats in a simple deterministic fashion. If you consider the myriad file encodings these days, you'll quickly realize that there is no uniform way to accurately categorize them all, because they're not all self descriptive (nor are they intended to be), and there's certainly no man-in-the-loop to open up them all and categorize them one by one.

What's needed is a uniform layer of descriptive information that's stored on a higher level than the file's actual contents. We call this higher level of information "data about data", or metadata (bust that word out on your next date).

When you first upgrade to Tiger, you'll notice that Spotlight takes some time to "index" your disk. During this process of indexing, Spotlight is sifting through your files to build up its metadata repository. This metadata consists of things like timestamps for when files were last opened, modified, and created; keywords in the files; the content types of the files; and so on. The metadata built up during indexing is pretty impressive, and you can further enhance it with "Spotlight Comments" (more on this in a moment). And before you ask, anytime you create or modify a file that Spotlight has indexed, it automatically updates its database and keeps current.

Overall, the process Spotlight uses for searching is similar to the way these modern marvels called search engines do their jobs. This is a great thing because search engines are the result of tons of research and development and do a tremendously better job at search than the way operating systems have traditionally handled the problem (recursively searching directories and system caches to scan filenames for keywords). For comparison purposes, Apple boasts that the search times using Spotlight are about 20 times faster than ordinary searching in Panther-based SearchKit. After running a few simple tests, I get comparable results.

If you're itching for more technical details before moving on to some specific ways Spotlight will make you productive, take a look at this Spotlight Overview or this supplemental information. There's also a pile of info out there on Google Desktop for Windows if you want to draw some comparisons between it and Spotlight. Finally, have a look at FSLogger if you want to really get insight into some of Spotlight's internals.

Specific Ways Spotlight Makes Your Life Easier

From Your Desktop

In the pre-Tiger days of not so long ago, I often found myself staring at my desktop trying to remember where I stored a file. I tend to meticulously organize files in multiple, nested folders, so sometimes this can be a chore--especially if I goofed and mistakenly saved it somewhere I didn't mean to. Sometimes, I'd find my file after navigating through several layers of folders in Finder. Other times I'd give up and rely on Finder to search for it while I side-stepped to check my email or find another way to momentarily distract myself. These techniques all took anywhere from three seconds to several minutes depending on the circumstances.

Turn to the little blue Spotlight icon up in your menubar for all of your searching needs. Turn to the little blue Spotlight icon in your menubar for all of your searching needs.

Now that I have Spotlight, I usually don't even bother navigating the hierarchy to look--even if I positively remember where a file is located. Instead, I click on the Spotlight icon (little blue icon) in my menubar, type what I recall the filename or general subject matter to be, and almost instantly get back what I expect. This is pretty cool because it requires less thought, is faster than I would be, and provides me with a single outlet for that instant gratification that I'm looking for.

Spotlight organizes your search results into categories using an arbitrary ordering by default, so I customized it in System Preferences to return PDF documents, followed by Documents, followed by Mail Messages, and so on, because this better suits my expectations. If you tend to work with certain file types more than others, you'll want to do a similar customization. For example, a graphic designer would want to have images and videos show up first, while an administrative assistant may want to have contacts and appointments show up first. Customizing your search results is just one additional way to increase laziness and productivity. Those two things usually don't come together so easily.

Order your search results by dragging and dropping categories to better suit your workflow. Order your search results by dragging and dropping categories to better suit your workflow.

When Saving Documents

One of the biggest annoyances is when you're already working in an application, need to have the application save a document, and find yourself navigating that hierarchy of file folders in the save panel. Life just got a lot better, because now you can save the double-clicks and use the search bar in the save panel to find that folder for you. You can also use the search box in the open panel, but I find this feature more useful for saving documents, since I tend to do most of my opening from the desktop.

Save time (and your patience) by not navigating through nested folders when saving a document Save time (and your patience) by not navigating through nested folders when saving a document.

With Safari Bookmarks

Like file folders, I organize bookmarks hierarchically--sometimes so hierarchically that it's annoying to open up some of them. Previously, I'd navigate directly from the "Bookmarks" menu and find myself holding down my mouse button while scrolling all over the place with my touch pad, often scrolling off of the menu and having to start over. The problem of sometimes misplacing a bookmark happened more often than I'd like to admit and occasionally complicated the matter.

Using the Spotlight icon in the menubar, I now retrieve bookmarks the same way I would retrieve any other file. I find this method preferable to Safari's "Collections" tab because it is generally faster. With a single click on the Spotlight icon the cursor is ready for you to type and search. Using the Collections tab in Safari, you have to actually click on the Collections icon, and then navigate to the bottom of the window and click on the search bar since it doesn't automatically take on the cursor--and this all assumes that Safari is already open and ready to go. Make every second count, and impress that special someone with your power-using self. Remember: real power users measure time in mouse clicks.

With a single click, look up bookmarks. With one more click, go there. With a single click, look up bookmarks. With one more click, go there.

With Advanced Searching in Finder

While the Spotlight icon in your menubar can do the job most of the time, there will still be those occasions when you want to do a more customized search. For these situations, you'll want to use Finder. Open up Finder's "File -> Find..." menu and you'll be presented with a fairly self-explanatory search window. Using this window, you can specify as many detailed criteria as you'd like. If you find that you've woven up a useful search you're likely to reuse, you can conveniently save it as a "smart folder" (more on this below).

For those highly customized search jobs, use Finder to make life easier. If you use certain searches often, you can save them as For those highly customized search jobs, use Finder to make life easier. If you use certain searches often, you can save them as "smart folders."

With Smart Folders

One of the coolest things about the way you can organize your music in iTunes is the concept of the "smart playlist." You can specify a particular artist, genre, album, and so on, and the folder automatically populates and updates itself without any further action on your part.

Spotlight takes this concept and expands it to a much broader level. Smart folders provide you with an always current collection of files that meet whatever constraints you want to establish. Uses for smart folders can be as creative as you can be. You can do anything from create a folder containing everything related to your kid's last birthday party (see more on "Spotlight Comments" below) to create a folder that contains only the rich text files that have been opened in the past two days. To create a Smart Folder, click on Finder's "File -> New Smart Folder" menu.

Smart folders are a really handy way to keep a collection of files current. Smart folders are a really handy way to keep a collection of files current.

With Burn Folders

Back in the olden days burning a data CD wasn't that hard, but it could still have been easier. Spotlight and burn folders make this task about as simple as it can be. You simply create a new burn folder via Finder's "File -> New Burn Folder" menu, type a search criteria into the search box and then wait a second for the results. When you get your results, you'll notice that the burn folder window has mysteriously transformed into a search result window. While that may be a bit inconvenient, it's really not a problem. Just select the files you want and drag them onto the burn folder that was created on your desktop (using Exposé to clear the desktop if need be). Ctrl-click or double click on the folder and then choose to burn the disc.

If you have a CD player capable of playing native MP3 or Windows media files in your vehicle, you'll find burn folders to be a handy way to take music with you (although DRM can make this less convenient than it could be). They're also useful for burning specific collections of files on multiple occasions.

Using Contextual Menus

While we've reviewed some of the more common ways that Spotlight will change the way you work, there are gobs of lesser known ones that we'll all continue discovering as time marches on. A miscellaneous use that I find convenient is to select some text in a Spotlight-aware application like Mail or Safari and then ctrl-click and choose to "Search in Spotlight" from the contextual menu. You might find yourself often doing this in Mail regarding work-related stuff.

By Adding Spotlight Comments

A final note of interest involves adding additional search criteria to Spotlight files. If you choose to "Get Info" on any Spotlight-aware file type or folder, there's now a text box entitled "Spotlight Comments" that you can use to provide Spotlight with additional keywords to help focus your queries.

If you find you need to add the same comments to many files, your best option is probably to create a smart folder and add the comments to the entire smart folder instead of to all of the individual files, but you have at least two other options: 1) get information on a selection of files to open up a screen full of tiled windows, and then use shortcut keys to paste to and close the windows in succession (if you're really scared of command lines), or 2) use command-line tools such as mdimport (check the man pages) to automate the process.

Give Spotlight additional keywords to further customize searching. Give Spotlight additional keywords to further customize searching.

Privacy Issues

You might find that Spotlight indexes content that you didn't want to be indexed--directories that contain personal information or things that you just don't want Spotlight being aware of. No problem. Just tell Spotlight not to index these directories via Spotlight's "Privacy" tab in System Preferences. If you find that you have the opposite problem, your easiest option is probably to use the mdimport -f option to force indexing of a directory. If you find the need to use a more sophisticated option, read this discussion.

Prevent sensitive information from being indexed by using Spotlight's privacy tab. Prevent sensitive information from being indexed by using Spotlight's privacy tab.

A Word of Caution

For your own awareness, you should realize that Spotlight makes it easy to do some serious snooping. For example, a smart folder created to display all files opened or modified within a specified time period can tell you an awful lot about what a person has been doing, when they've been doing it, and so on. With almost no effort, you can basically reconstruct an entire day's workflow to include everything from what music a person listened to, to which files they modified and created.

You owe it to yourself to take a moment to search your own machine for "social security number," "birthday," or similar phrases you might consider sensitive. Ever heard of getting "Google hacked"? Getting "Spotlight hacked" isn't that much of a stretch. If you want to be on the safe side, don't allow Spotlight to unknowingly abet a malicious script designed to harvest your personal info. These kinds of things could happen--and a completely unconstrained Spotlight sure would make a great accomplice. At least think about it.

Go and Be Productive

Spotlight and the new features that it brings to Tiger are bound to save you time and make you more productive. Go get some work done using these new features, and stop worrying about the pains that used to come with finding files in your uber-nested folders--or maybe it was only me that had that problem?

Matthew Russell is a computer scientist from middle Tennessee; and serves Digital Reasoning Systems as the Director of Advanced Technology. Hacking and writing are two activities essential to his renaissance man regimen.


Return to MacDevCenter.com.

Copyright © 2009 O'Reilly Media, Inc.