What's New For Developers in QuickTime

QuickTime 7 has made a splash since premiering as part of Mac OS X 10.4 (Tiger) and as a separate download for users still on 10.3 (Panther). In Part 1 of this series, I covered the features of QT7 that end users will see, specifically:

• Expiration of QuickTime Pro 6 keys (maybe not a "feature", per se...)
• One-touch audio and video recording menu items in QuickTime Player
• New properties and controls windows in QuickTime Player
• The H.264 video codec
• Support for authoring multichannel audio
• A new, web-based QuickTime Component Download Program

In this second article, I'll cover some of the most prominent changes from a developer point of view. These include:

• The QTKit API for Cocoa
• The new QuickTime MetaData API
• Support for frame-reordering codecs
• Core Audio-based sound enhancements
• Miscellany: JavaScript and QuickTime for Java

Programming with QTKit

If, like me, you've picked up Cocoa and Objective-C after spending years in other object-oriented languages like Java, you probably don't mind the differences in syntax—you say square brace, I say dot operator—because the ideas are fundamentally similar. And to be sure, developing Cocoa applications is quite pleasant when you get to use tools like Interface Builder, more pleasant than writing hundreds of lines of GridBagLayout code, like us J2SE guys are used to.

Unfortunately, Cocoa access to QuickTime functionality has been limited up to now. Two Application Kit classes, NSMovie and NSMovieView, provided a bare-bones ability to open movies from the clipboard or a URL and to send them a few basic messages: start and stop; cut, copy, and paste; setEditable and isEditable, etc. Deeper needs than this required taking a pointer to the QuickTime Movie and heading off to the straight-C QuickTime API.

QuickTime 7's SDK—installed by Tiger or via Software Update if you have XCode Tools installed on Panther—includes a new framework called QTKit, which significantly increases the number of things you can do in Cocoa and Objective-C, without needing to resort to more difficult C calls to QuickTime itself.

QTKit is still fairly small, consisting of just five classes:

• QTMovie: a collection of media resources and their relative arrangement.
• QTMovieView: a visual component for displaying movie contents and (optionally) a controller widget.
• QTTrack: an organization of one kind of media, describing its timing, spatial, layering, volume, etc.
• QTMedia: a reference to actual samples of some kind of media—frames of video, clips of text, samples of audio, etc. Each track has exactly one media.
• QTDataReference: a specification of the location of some media data, potentially in memory, on disk, or on the network.

You might be tempted to think that QTMovie and QTMovieView would be simple replacements for NSMovie and NSMovieView, but the docs reveal a surprise. While NSMovie offered just 11 methods, QTMovie offers 77, including some shockingly powerful calls. For example, addImage:forDuration:withAttributes: effectively allows you to build video media, frame by frame, without even looking at the compression sequence API.

QTKit has its limits, of course. There's no object to represent the SequenceGrabber for media capture, nor a wrapper to the Presentation for broadcasting. Furthermore, the objects that do exist don't wrap every possible QuickTime call with which each object might be used. Think of it as an 80-20 proposition: what's provided so far is a small subset of the QuickTime API, the most popular calls that will allow many developers to meet their needs without delving into the straight-C API.

A programming example of QTKit appears later in this article, after covering another new feature.

Meeting Metadata

Related Reading QuickTime for Java: A Developer's Notebook By Chris Adamson

The ability to "tag" media with descriptive data is becoming essential for supporting search, as well as providing a better end-user experience. However, schemes for tagging media are all over the map: the ID3 standard for MP3's, EXIF for JPEG's, and an undocumented and proprietary format for iTunes-encoded AACs (that I investigated and provided a parser for in the ONJava article QTJ Audio and in the book QuickTime for Java: A Developer's Notebook). To top it off, QuickTime movies allow for a user data container, into which you can place pretty much any data you like.

Implementing support for all of these formats in your application would be a headache.

Realizing this, QuickTime 7 provides a new metadata API that abstracts away the differences between the many schemes, exposing all of them through a common interface. In its first implementation, QuickTime supports three "containers":

• Classic QuickTime user data: along with supporting genuine QuickTime user data, this also provides access to MP3s' ID3 tags and other forms of metadata that are parsed into user data items when files and streams are imported into QuickTime.
• iTunes Metadata: album, artist, song, user rating, cover art, etc., from files encoded by the user or purchased from the iTunes Music Store. Note that according to Movies.h, "The list of keys for iTunes metadata is TBA."
• New QuickTime metadata storage format: the docs advertise this as simply "a richer QuickTime metadata container format." Among its most interesting traits is the abandonment of the four-character-code scheme of user data keys in favor of a reverse-DNS format, similar to package names in Java (e.g., com.apple.something.somethingelse). This has the advantage of avoiding name collisions, not requiring Apple to act as a universal registrar of key names, and being infinitely extensible.

The idea is one of simple key-value coding: you supply a known key (like "description" or "album name"), and you get back 0 or more values. In practice, using the Metadata API has significant similarities to getting Components from QuickTime: you search for them by providing criteria to match against, you call the search method with the last match (null on the first call) to find the next match, and there can be several matches even when asking for fairly specific criteria. To clarify the last point, you can ask for a given key in a given container format and get multiple hits. After all, there might be multiple "author" or "artist" tags.

Demo: QTKit and Metadata in Practice

By way of a demo, here's a short Cocoa application to show:

• How to use QTKit's QTMovieView in a nib
• How to get a QTMovie and put it into the QTMovieView
• How to move from QTKit to QuickTime, by getting the underlying Movie pointer from a QTMovie object
• How to get metadata from the movie with the new metadata API

I'll walk through all the Xcode steps, but this is going to be lightning fast; for a proper introduction to Cocoa development, check out James Duncan Davidson's Learning Cocoa with Objective-C, 2nd Edition. The complete Xcode project is available in the resources at the end of the article.

Building the QTKit GUI

First, kick off a new Cocoa-application project in Xcode. Double click the MainMenu.nib to launch Interface Builder (IB). The first thing you need to do in IB is to add the QTKit palette, so that you'll be able to add a QTMovieView to your GUI. So, do a "Show Palettes" (if the Palettes window isn't already up) and go to "Palette Preferences..." Click the "Add..." button and navigate to /Developer/Extras/Palettes/QTKit.palette to add the QTKit palette. You should see the QTKit palette in your Palettes window, as seen in Figure 1.

Figure 1. QTKit Palette in Interface Builder

The big "Q" is a QTMovieView. Drag it to the main window to add a QTMovieView to your GUI. It will originally appear as a very small component; I used the Inspector to set its size to 320 x 256 (the common 320 x 240 movie size, plus the usual 16 pixels for the scrubber bar) and resized the window to align it to the bounds suggested by IB's friendly blue lines. The resulting window should look like Figure 2.

Figure 2. QTMovieView in an IB window

Finish off the window by adding an NSTextView at the bottom and a label above it that says "Metadata".

Now you need to start wiring things up, by creating a controller object to handle UI events. In the MainMenu.nib window, click the Classes tab, and then click NSObject. Go to the Classes menu and do "Subclass NSObject". Rename the new class MyController.

Now tell the controller about its actions and outlets: with MyController selected, bring up the inspector. Add two outlets, naming them metadataView and movieView. Also add an action, which you can call handleOpenMovie.

You need an instance of the controller, which you'll wire up to the GUI components. With MyController selected, pull down "Instantiate MyController" from the Classes menu. This will create a "MyController" object, represented as a blue cube in the Instances tab of the MainMenu.nib window. To wire up the outlets, control drag from the object to the QTMovieView, as illustrated in Figure 3. This will bring up the inspector, and allow you to select which of MyController's outlets you're connecting. Choose movieView. Now repeat the process to wire up the metadataView to the NSTextView.

Figure 3. Wiring up a QTMovieView as an outlet

In the MainMenu.nib window, open the MainMenu object and trash all the menus except "File", and delete all of its menu items except for open. Control drag from the open menu item to the MyController object to wire up the menu item to the handleOpenMovie action.

Everything is now wired up. In the MainMenu.nib window, switch to the "Classes" tab and select "MyController". Go up to the "Classes" menu and select "Create classes for MyController". The default location should be good, and you should see that "insert into targets" says "QT7MiniDemo" (or whatever you named your project). Save up and quit IB.

QTKit Coding

Back in Project Builder, your next job is to add in the QTKit and QuickTime frameworks, so you can compile code against them. Control click on "Linked Frameworks" and choose "Add -> Existing Frameworks...". Navigate to /System/Library/Frameworks and choose QTKit.framework. Then do the same thing to add QuickTime.framework. You'll also need to import these frameworks in your header file, so open MyController.h and add the lines:

    #import <QuickTime/Movies.h>
    #import <QTKit/QTKit.h>

Next, you should strongly type the outlets listed in Movies.h so the compiler can better check your code. Your outlets should look like this:

    IBOutlet NSTextView *metadataTextView;
    IBOutlet QTMovieView *movieView;

All right, now you're ready to code! In the handleOpenMovie method, your job is to present a file dialog, open the selected file as a QTMovie, and put the movie in the QTMovieView. The first part is pretty easy... OK, it's all pretty easy, but this part you might have seen before:

NSArray* fileTypes = [NSArray arrayWithObjects:@"mov", @"mp4",
                        @"m4a", @"mp3", @"m4p",
                        @"jpg", @"jpeg",nil];
NSString* moviesDir = [NSHomeDirectory() stringByAppendingString:
                        [NSString stringWithString: @"/Movies"]];
// ...
    NSOpenPanel* panel = [NSOpenPanel openPanel];
    [panel runModalForDirectory:moviesDir file:nil types:fileTypes ];
    NSURL* url = [[panel URLs] objectAtIndex: 0];

This just shows an NSOpenPanel for several known QuickTime-friendly file types. When the panel is dismissed, you get the selection as a URL.

Next job: converting this file into a QuickTime movie. Hold on, this is going to be tough...

    NSError* openError = nil;
    qtMovie = [QTMovie movieWithURL: url error:&openError];

...or not. Getting a QTMovie from a URL is a one-line call. OK, a few more lines if you want to inspect the error if it isn't nil; this is in the downloadable code, but I've omitted it here. Assume for now that it worked and that you have a valid QTMovie object. Now you need to put it in the view:

    [movieView setControllerVisible: YES];
    [movieView setMovie: qtMovie];

And... we're done. Seriously. This is all you need to do to open and show a QuickTime movie with QTKit. The control bar is live, so you don't even need to provide menu items or buttons to start and stop movie playback, though it's not like a [qtMovie start]; would exactly kill you, now would it?

This is an extremely simple example of QTKit, practically the "Hello World" of using the framework. For a far more ambitious example, check out Apple's tutorial QuickTime Kit Programming Guide.

One little note here: I've noticed that when I do a development build, the QTMovieView doesn't become active until I switch to another application, and then back to QT7MiniDemo. This doesn't happen when I build the application via a deployment build. Just so you don't think you made a mistake...

Moving on to Metadata

As if to remind you how pleasant QTKit is, this next section on metadata requires using the straight-C QuickTime API. First, get a pointer to the QuickTime movie:

    Movie movie;
    movie = [qtMovie quickTimeMovie];

The first thing to do with metadata is to get a reference to a QTMetaDataRef. You can get this from a movie, any of its tracks, or any of their media; obviously, this parallels the use of QuickTime user data, which exists at each of those levels. For the purposes of this demo, you're only interested in Movie-level metadata:

    QTMetaDataRef movieMetaData = malloc (sizeof (QTMetaDataRef)); 
    QTCopyMovieMetaData (movie, &movieMetaData); 

This object will give you metadata one of two ways: you can either ask for specific metadata items by known keys, or you can use wildcards to tour the available metadata. You need to specify which kind of container you want to use (user data, iTunes, new QT metadata), but there's a wildcard for that too. You retrieve a QTMetaDataItem by way of the QTMetaDataGetNextItem function. As you might have guessed from the use of "Next" in the method name and other QuickTime discovery conventions (like FindNextComponent), you iterate through the metadata items by passing in the last one you found, using kQTMetaDataItemUninitialized for your first trip through the loop. This makes for a loop that looks like the following:

    QTMetaDataItem item = kQTMetaDataItemUninitialized;
    while (noErr == QTMetaDataGetNextItem (movieMetaData,
                                kQTMetaDataStorageFormatWildcard,
                                item,
                                kQTMetaDataKeyFormatCommon,
                                nil,
                                0,
                                &item)) {

        // do stuff with item

    }

This call is documented in the QT API reference, but to quickly summarize the parameters:

1. The QTMetaDataRef you obtained earlier
2. A constant for the container (user data, iTunes, new QT metadata) that you want to search in
3. The last QTMetaDataItem you found
4. A constant for the format of the key you're querying with
5. A pointer to the key you want to search by. Pass nil for a wildcard
6. The size of the key
7. A pointer to receive the address of the next matching metadata item.

Once you have a QTMetaDataItem, there are various things you can do with it. By using the QTMetaDataGetItemProperty method, you can get the key's type, size, and value. For this example, it's nice to show the keys that go with each value, so you'll know what to search for in the future. Here's how to get a key from an item discovered via wildcards:

    UInt8 buffy[100];
    // ...
    QTPropertyValuePtr key = &buffy;
    ByteCount keySize = 0;
    QTMetaDataGetItemProperty (movieMetaData,
                               item,
                               kPropertyClass_MetaDataItem,
                               kQTMetaDataItemPropertyID_Key, 
                               sizeof (buffy),
                               key,
                               &keySize);
    NSString* keyString =
        [NSString stringWithCString: key length: keySize];

As you can see QTMetaDataGetItemProperty takes the QTMetaDataRef and the QTMetaDataItem, along with a "class" and "ID" indicating what you want to get from the item. The last two parameters receive a pointer to the data and its size, from which you can easily make an NSString for later use with the GUI.

So you have a key. Next, you want a value. This can be potentially huge (think cover art in iTunes), so you precede the "get value" call with a "get size of value" call, in order to allocate a sufficiently large buffer:

    // get value size
    ByteCount valueSize = 0;
    QTMetaDataGetItemValue (movieMetaData,
                            item,
                            NULL,
                            0, 
                            &valueSize);
    printf ("  got value, size=%d\n", valueSize);
    // get value
    char valueBuf[valueSize];
    QTMetaDataGetItemValue (movieMetaData,
                            item,
                            &valueBuf,
                            valueSize, 
                            NULL);

As you can see, these are the same function, except that the first time, you pass in NULL for the second parameter (outValuePtr) to have the size returned in the last parameter. On the second call, you pass a pointer to a sufficiently large buffer in the second argument and ignore the last argument.

One problem left: what did you get back? If you query by a known key, you might know how the returned buffer is organized, but for a wildcard tour like this, you don't know what you got back. Fortunately, there's a call for this too:

    QTPropertyValuePtr* valueType = buffy;
    ByteCount valueTypeSize = 0;
    QTMetaDataGetItemProperty (movieMetaData,
                               item,
                               kPropertyClass_MetaDataItem,
                               kQTMetaDataItemPropertyID_DataType, 
                               sizeof (buffy),
                               valueType,
                               &valueTypeSize);

The returned QTPropertyValuePtr can be one of the following types defined in Movies.h:

  kQTMetaDataTypeBinary         = 0,
  kQTMetaDataTypeUTF8           = 1,
  kQTMetaDataTypeUTF16BE        = 2,
  kQTMetaDataTypeMacEncodedText = 3,
  kQTMetaDataTypeSignedIntegerBE = 21, 
  kQTMetaDataTypeUnsignedIntegerBE = 22,
  kQTMetaDataTypeFloat32BE      = 23,
  kQTMetaDataTypeFloat64BE      = 24

In the example, these values are used to determine whether the value can be converted into a string (ultimately into an NSAttributedString) and displayed in the NSTextView. If not, an array of type names allows the view to show a description of the discovered value.

That's pretty much it. If you have QuickTime Pro, you can test it by using the Properties viewer to add some annotations to a Movie. Figure 4 shows an example of a movie I've annotated this way.

Figure 4. QT7MiniDemo showing Movie-level annotations

If you don't have QuickTime Pro, don't panic. Lots of QuickTime files have interesting metadata that can be read with this application. For example, the metadata for an iTunes Music Store song is shown in Figure 5.

Figure 5. QT7MiniDemo showing iTunes Music Store metadata

One last talking point before moving on... do you suppose that Apple added this feature to QuickTime just to be nice? I don't. Most recent QuickTime features have been added to service strategic goals, like how AAC and Apple Lossless bolstered the iPod. Let me show you what I think is up here. If I use Spotlight to query for "Elk Rapids", a fairly unique term in my annotation, Figure 6 shows what I get:

Figure 6. Finding QuickTime metadata with spotlight

Notice that "Elk Rapids" isn't in this file name, only in its metadata. On the other hand, don't just assume that Spotlight searches the entire file. That would be incredibly pointless on a multi-gigabyte video file. Moreover, it's demonstrably not true: every QuickTime movie contains an atom called moov, and every tagged MP3 starts with the string ID3, yet searching for those strings doesn't pull up all your movies or all your MP3s. I think this suggests that Spotlight uses the QuickTime metadata API when indexing your movies, and I wonder if that's the reason the metadata API was developed to begin with. (Join me in the talkbacks below for further idle speculation.)

More New Stuff

As I'm already over the MacDevCenter article word limit, let me briefly discuss a few other points that will be of interest to specific classes of QuickTime developers.

Support for Frame-reordering Codecs

To support H.264, and other modern codecs, the image decompression API has seen significant changes to support "out of order" or "frame-reordering" codecs. These kinds of codecs allow for three kinds of video frames:

• Intra-frame (I-frame): a self-contained frame, i.e., one that does not depend on any others for decoding
• Predicted frame (P-frame): a frame that depends on one or more previous frames (an I-frame and zero or more other P-frames)
• Bi-directional frame (B-frame): a frame that depends on one or more previous frames and a future frame

Being able to depend on a future frame increases compression efficiency, but increases cost and complexity. On this latter point, imagine the set of frames illustrated in Figure 7:

Figure 7. Conceptual layout of I-, P-, and B-frames

In this example, the letters equal the frame type and the numbers indicate their order. So, frames 1 and 5 are independent, 2 depends on 1, 3 depends on 2 (and thus on 1), and 4 depends on 3 and on 5. That means to display frame 4, frame 5 has to be decoded first, even though it will be rendered after 4. This can result in an encoding that actually puts frame 5's data before 4's, as seen in Figure 8.

Figure 8. Example of a reordered B-frame

What this introduces in the API is a distinction between decode time and display time since some frames will be decoded but not used right away (as is the case with frame I5 above). Notes on CompressSequenceBegin and DecompressSequenceBegin in ImageCompression.h now advocate use of the alternatives ICMCompressionSessionCreate and ICMDecompressionSessionCreate. These require a change in coding style too, because instead of providing a frame in a return value or parameter, they use a callback function, and can return multiple frames.

Most application-level developers don't need to worry about these issues, but applications that compress or decompress media at the sample level, or that count samples, need to understand the concept and look for new functions that are frame reordering aware.

Audio Enhancements

QuickTime 7 makes a number of changes to improve audio support. A new SoundDescriptionV2 opens the way to multichannel audio (exploiting Core Audio on Mac OS X to support it... thus indicating it's not meaningfully supported on QuickTime for Windows), but you aren't responsible for actually dealing with this structure. Instead, a higher-level sound description API allows you to create the description with an AudioStreamBasicDescription, a channel layout, and (optionally), a "magic cookie." And yes, I said, "channel layout." QuickTime now goes beyond left-right stereo to support arbitrarily placed sound sources, such as 5.1 surround-sound arrangements.

Other small but novel improvements include the ability to change playback rate without pitch-shifting audio, and level and frequency metering API's. On this latter point, previous versions of QuickTime offered some level-metering functions at the media level, but they were undocumented.

Safari Support for JavaScript Access to the QuickTime Plug-in

Many popular browsers have been able to use JavaScript to control an embedded QuickTime movie for some time, but this feature now works on Safari in Tiger. This means that you can do some simple tricks, like providing script-based start and stop buttons as seen in this example from an ONJava QuickTime for Java article.

New Fixes, New Bugs in QuickTime for Java

Speaking of QuickTime for Java, it got a little love in QuickTime 7, with a number of outstanding bugs fixed for it. Unfortunately, QuickTime 7 does not address many long-standing requests from QTJ developers, like the ability to get an AWT or Swing preview component for the SequenceGrabber. Also, QT7 seems to have introduced some new QTJ bugs of its own: displaying the settings dialog for a streaming Presentation, something I discussed in the article Streaming QuickTime with Java, is now a 100% crasher. Sigh. QTJ giveth, and QTJ taketh away.

Big Finish

This article only broadly introduces the major changes in QuickTime 7; when I diff'ed the header files between it and QuickTime 6, I had over 9,000 new lines just of constants, functions, and comments. A more in-depth tour of QT7's major new features is available in Apple's QuickTime 7 Update Guide. Still, I hope it will get you thinking about the new things you can do with QuickTime in Tiger.

Resources

• Sample code for the QTKit/metadata application
• QuickTime 7 Update Guide
• QTKit Programming Guide

Chris Adamson is an author, editor, and developer specializing in iPhone and Mac.

Return to the Mac DevCenter

Copyright © 2009 O'Reilly Media, Inc.