MacDevCenter    
 Published on MacDevCenter (http://www.macdevcenter.com/)
 See this if you're having trouble printing code examples


Programming With Cocoa
Inside StYNCies, Part 2

by Matthew Russell
03/18/2005

The first installment of this dynamic duo developed a user interface that lives up in your menubar like the system clock. This piece finishes off the series by reverse-engineering the storage format of the StickiesDatabase file to develop your own API to Stickies and to finish off your project for syncing Stickies with your iPod.

A Quick Word About Reverse-Engineering

In recent times, a "hack" has been redefined as a clever solution to a problem. Dictionary.com, going in a different direction, defines a person who hacks as "One who uses programming skills to gain illegal access to a computer network or file." Let's be clear that this is something we're definitely not doing.

We're simply reverse-engineering a fairly well known file format in order to provide some interoperability. The file format is unencrypted, and its data could actually be parsed out using regular expressions, but that would be very crude and there wouldn't be a lot to learn. So instead, we'll use some standard Cocoa tools to develop a nice API.

Before you ever reverse-engineer any file format or unpublished specification, you must realize that reverse-engineering is a double- edged sword: On the one side, it gives you the ability to exploit the utility of an application to provide exciting interoperability features with other apps that aren't currently there. On the other side, the next operating system update or release might provide an update that breaks your API, and so you have to track it down and fix it. Being mindful of these things, let's have some fun.

Essential Tools

The Rosetta Stone provided the key to Egyptian hieroglyphics back in 1799. We don't have a Rosetta Stone, but we do have some other handy tools and clues for unscrambling the StickiesDatabase file format. Namely, our trusty text editor VIM, the class-dump utility, and some of Apple's developer documentation.

If you don't have class-dump installed already (type "class-dump" in Terminal to check), you can download it from here. VIM comes packaged with OS X, and Apple's developer documentation is easily found in Xcode's help menu. Take a moment to preview the usage info for class-dump as well as Apple's documentation for NSUnarchiver before moving on.

You'll need to backup the StickiesDatabase file found in your ~/Library/ directory for reasons you'll understand in just a moment. In Terminal, you can type "mv StickiesDatabase StickiesDatabase.bak" to rename it. When you open up Stickies again, you'll see the three default notes. Close two of them without saving and leave only a few words of text in the remaining note. Close Stickies. With VIM or another text editor, open up the "new" StickiesDatabase file and look at it. Having a very simple file to work with makes the entire reverse-engineering process much easier than dealing with multiple notes with all sorts of things embedded in them.

StickiesDatabase in VIM The StickiesDatabase file in VIM. The database consists of a single yellow sticky note with the text "A FEW WORDS" written on it.

Some Detective Work

From peeking at your StickiesDatabase file, it's obvious that it's not a textual plist file (type "man plist" in Terminal for more info on the plist format). If you've done any Cocoa programming before, you know that NSUnarchiver and its new counterpart NSKeyedUnarchiver are the standard Cocoa ways of reading data from files, so these two classes are good places to start.

NSArchiver and NSUnarchiver make it really easy to write and read data to disk. Each class that is to be archived simply implements the NSCoding protocol. Objects that take care of this detail can be stored individually to disk or in other archivable data structures that also implement the protocol. If you were to guess, how would you think multiple sticky notes would be stored to disk? NSMutableArray sure does seem plausible. Remember seeing the words "NSMutableArray" as plain text in your StickiesDatabase file? Knowing that NSArray inherits from NSMutableArray, and NSMutableArray inherits from NSObject, look again at StickiesDatabase in VIM, and take a stab about the structure of the file format.

From reading the documentation, you know that unarchiving data with NSUnarchiver requires calling its unarchiveObjectWithFile: method on an object whose constituent subparts are also unarchivable. So if you were to unarchive a file such as ~/Library/StickiesDatabase containing an NSMutableArray filled with archivable objects, you'd simply do it as so:

NSMutableArray *array = 
        [[NSMutableArray alloc] initWithArray:
                [NSUnarchiver unarchiveObjectWithFile:
                        [@"~/Library/StickiesDatabase" 
                                stringByExpandingTildeInPath]]];

Again, in order for unarchiveObjectWithFile: to be successful, all of the objects encountered during the unarchive process have to actually exist, be of the correct form, and implement the NSCoding protocol. To be of the correct form, the objects must have the correct variables declared in their interface. Sound like a chore? Well, class-dump helps a lot.

Inside the Stickies Application Bundle, there's a Mach-O executable. Running class-dump on it provides all of the interface details we need. In Terminal, type "cd /Applications/Stickies.app/Contents/MacOS" and you'll find the Stickies executable there by typing "ls" to get a directory listing. Run class dump on it and save the output to a file on your desktop like so: "class-dump Stickies > ~/Desktop/Stickies.dump". Take a moment to look at the output with VIM or TextEdit. It's pretty neat. The part we're most interested in is inline below.

//Part of the output from running class-dump
//on the Stickies Mach-O executable.

@interface Document : NSObject <NSCoding>
{
    int mWindowColor;
    int mWindowFlags;
    struct _NSRect mWindowFrame;
    NSData *mRTFDData;
    NSDate *mCreationDate;
    NSDate *mModificationDate;
}

- (id)init;
- (void)dealloc;
- (id)initWithCoder:(id)fp8;
- (id)initWithData:(id)fp8;
- (void)encodeWithCoder:(id)fp8;
- (id)creationDate;
- (void)setCreationDate:(id)fp8;
- (id)modificationDate;
- (void)setModificationDate:(id)fp8;
- (id)RTFDData;
- (void)setRTFDData:(id)fp8;
- (int)windowColor;
- (void)setWindowColor:(int)fp8;
- (int)windowFlags;
- (void)setWindowFlags:(int)fp8;
- (struct _NSRect)windowFrame;
- (void)setWindowFrame:(struct _NSRect)fp8;

@end

There are a couple of things worth mentioning at this point. One is that there's a lot of user interface stuff that goes into any nice Cocoa application; Stickies is no exception, despite its simplicity. Another thing is that the Document class is the only one that implements the NSCoding protocol. This is good news for us because this interface is all that we need to implement in order to unravel the mystery of the StickiesDatabase file. Do you remember seeing the word "Document" in the StickiesDatabase file? By this point, you've most likely realized that NSArchived files contain plain text names for all of their archived classes. The data in these classes, however, is another story, as we will see.

Learning Cocoa with Objective-C

Related Reading

Learning Cocoa with Objective-C
By James Duncan Davidson, Apple Computer, Inc.

Defining the Implementation for "Document"

To clarify, the Document class has no relation to NSDocument, although Stickies is indeed a document-based application. "Document" in our context is just another word for "sticky note." If you make a good guess right now, I bet you'd venture to say that the answer to our NSUnarchiver riddle is that an NSMutableArray containing items of type Document is what is stored in the StickiesDatabase file. Let's fill in the implementation to see if this is correct.

Let's get going in Xcode:

You'll notice that the vast majority of the methods appear to be trivial to implement, because Document is Key Value Coding (KVC) compliant. We know that initWithCoder: and encodeWithCoder: are part of the standard NSArchiving paradigm, and init: and dealloc: are standard methods for any Objective-C class. That leaves only initWithData: to figure out, which turns out to be pretty simple. Objective-C design principles require that each class have a designated initializer. This means that initWithData: is probably the designated initializer, and init: simply calls initWithData: to pass in a default value. Even if this isn't the way it's actually done by Apple, it turns out to work for us.

If you're a real computer nerd, you know that there's a countable infinity of ways to reconstruct any program using only the "black-box" approach that we're using, so don't get too hung up on doing it exactly the way Apple does it. There's no way to ever know aside from looking at the actual assembly code and reconstructing it. Even then, the compiler most likely optimized it. For some hard (but very interesting) reading that gets at the heart of this issue and relates it to other parts of life in ways you may not have expected, check out the book Gödel, Escher, Bach: An Eternal Golden Braid.

Getting back on track, the interface from class-dump lists all method return values and arguments as type id and gives structs as their literal naming from inside the Cocoa frameworks. We can take care of both of these issues easily. Simply replace "struct _NSRect" with "NSRect" and take your best guesses as to the KVC naming for method return types and arguments (or see the code below). The KVC naming conventions make it easy. It's also helpful to rename the arguments to something that makes more sense than "fp8."

The code runs if you don't actually do any of this, but if you don't, you have to define and refer to your own local "_NSRect" instead of Cocoa's version of the same thing. Statically typing the arguments and return types sometimes helps the compiler to warn you when you do something you probably didn't intend to do. Below is a partial implementation of the Document class. The methods that are needed for archiving Document objects to disk don't have bodies defined, because our project is limited to reading from the StickiesDatabase file.

If you're interested in getting the full reverse-engineering experience and are willing to kill a lot of time, try scrambling the loading order up in initWithCoder: and/or use some different unarchiving methods in each step. Work out the kinks based on the error messages that you get instead of just using the given solution below. If you just want to get this app working, a simple cut and paste works fine.

//probably sets some reasonable defaults
//and calls the designated initializer
- (id)init {
        NSLog(@"^^^init:");
        NSMutableDictionary *atts = 
                         [[NSMutableDictionary alloc] init];
        [atts setValue:[NSFont fontWithName:@"Helvetica" size:0.0] 
                forKey:NSFontAttributeName];

        NSAttributedString *attStr = [[NSAttributedString alloc] 
                initWithString:@"" attributes:atts];

        [attStr autorelease];
        [atts autorelease];
 [self initWithData:[attStr RTFDFromRange:NSMakeRange(0, 11) 
                documentAttributes:nil]];
}


//probably the designated initializer
- (id)initWithData:(NSData*)d {
        NSLog(@"^^^initWithData:");
        /* TODO: if we want to write to StickiesDatabase */     
        return self;
}

- (void)dealloc {
        NSLog(@"^^^dealloc:");
        [mRTFDData release];
        [mCreationDate release];
        [mModificationDate release];
        [super dealloc];
}

//unarchive
- (id)initWithCoder:(NSCoder*)c {
        NSLog(@"^^^initWithCoder:");
        [super init];
        mRTFDData = [[c decodeObject] retain];
        [c decodeValueOfObjCType:@encode(int) at:&mWindowFlags];
 [c decodeValueOfObjCType:@encode(NSRect) at:&mWindowFrame];
        [c decodeValueOfObjCType:@encode(int) at:&mWindowColor];
        mCreationDate = [[c decodeObject] retain];
        mModificationDate = [[c decodeObject] retain];
        return self;
}

//archive
- (void)encodeWithCoder:(NSCoder*)c {
        NSLog(@"^^^encodeWithCoder:");
        /* TODO: if we want to write to StickiesDatabase */
}
- (NSDate*)creationDate {
        return mCreationDate;
}
- (void)setCreationDate:(NSDate*)d {
        [d retain];
        [mCreationDate release];
        mCreationDate = d;
}
- (NSDate*)modificationDate {
        return mModificationDate;
}
- (void)setModificationDate:(NSDate*)d {
        [d retain];
        [mModificationDate release];
        mModificationDate = d;
}
- (NSData*)RTFDData {
        return mRTFDData;
}
- (void)setRTFDData:(NSData*)d {
        [d retain];
        [mRTFDData release];
        mRTFDData = d;
        [self setModificationDate:[NSDate date]];
}
- (int)windowColor {
        return mWindowColor;
}
- (void)setWindowColor:(int)c {
        mWindowColor = c;
        [self setModificationDate:[NSDate date]];
}
- (int)windowFlags {
        return mWindowFlags;
}
- (void)setWindowFlags:(int)f {
        mWindowFlags =f;
        [self setModificationDate:[NSDate date]];
}
- (NSRect)windowFrame {
        return mWindowFrame;
}
- (void)setWindowFrame:(NSRect)f {
        mWindowFrame = f;
        [self setModificationDate:[NSDate date]];
}

Being mindful of memory management, we use the "Retain, then Release" approach. The method initWithCoder: is fairly straightforward if you have it there in front of you, but involves some good guesswork to get right and is the "time sink" of the endeavor to read from the StickiesDatabase. NSCoder's decodeValueOfObjCType:at: method is worth taking a look at in some depth. There's more than one way that data can be decoded and encoded; this just happened to be the one that worked out.

You should also notice that just because the data loads without NSUnarchiver blowing up on you, that still doesn't mean that you have the logic correct. It simply means that you specified objects in the correct ordering. For example, you can switch mWindowFlags and mWindowFrame (both are of type int) and the data loads fine. You pare down issues like these with trial-and-error, and by looking at values you know to be correct in the debugger or with NSLog statements.

The Color menu in Stickies
The integer constants used for specifying the colors of notes in the Document class correspond to the same indexes of options from the "Color" menu in Stickies. Use these types of clues to make your work easier.

For example, you can determine the color and time stamps of your really simple StickiesDatabase file that has only a single note in it by opening up Stickies and looking at the tooltip that appears when you hover your mouse over the note. Using this information, you can rearrange your loading order in initWithCoder: to get things straightened out if you log and inspect the output of each step.

For purposes of syncing with your iPod, you might want to stamp each note with the creation/modification date, but the color of the note and window flag are not relevant for iPod use. Check out your skills, and see if you can define all of the colors and window flags as constants in your Document class using this procedure. At the end of this article is a link to a full implementation of the Document class that gives them to you, but this is still a good exercise to try on your own.

The method name and instance variable containing "RTFDData" gives us a big hint as to the type of data stored in a note. If you've used Stickies enough you've probably already realized that they store Rich Text Format (RTF) data. Choosing to export notes via the "File" menu in Stickies reveals the RTF and RTFD output options.

Export options in Stickies It's no coincidence that the export options for sticky notes correspond to the RTFD types seen from the class-dump output (or that there are extensive developer classes for working with RTFD).

Files of type RTFD (Rich Text Format Directory) are like application bundles; they're actually a directory that contains goodies like images or other embedded media referenced in an accompanying RTF file. Export some notes containing images or other embedded media as RTFD and go take a look. There's a formal RTF specification written up about RTF documents, and Apple documents their RTF extensions in the relevant portions of their own documentation. If you skim through any of it, you'll discover that RTF is just plain text with fairly incomprehensible formatting markup added. In the simple StickiesDatabase file you looked at in VIM, about half of what you saw was actually RTF data stored as plain text.

There's a lot of different ways you could go to parse the RTFD data. Since we're simply writing text to the iPod, however, we have it easy. (If you wanted to go in some other directions, you could start by looking here.) We use a class called NSAttributedString that can initialize itself with RTF data, so it's then a simple matter then to convert the data to an attributed string and then extract only the plain text from the string. Add the following code to your Document class. These are convenience methods that have no impact on our ability to unarchive stickies; only a class's instance variables matter as far as NSUnarchiver is concerned.

//add these declarations to Document.h
- (NSString*)stringValue;
- (NSString*)documentTitleOfLength:(int)length;


//add these bodies to Document.m
- (NSString*)stringValue {
        NSAttributedString *str = 
                [[NSAttributedString alloc] 
                        initWithRTFD:[self RTFDData] 
                        documentAttributes:NULL];
        
        [str autorelease];
        return [str string];
}

- (NSString*)documentTitleOfLength:(int)length {
        //the first "length" chars may contain control chars 
        //like newline. These don't look so good as filenames.
        //Take either the first "length" chars
        //or chars until a control char is reached
        NSString *t;
        
        if ([[self stringValue] length] >= length)
        {
                t = [[self stringValue] substringToIndex:length];
        }
        else
        {
                t = [self stringValue];
        }
        
        NSRange controlRange = [t rangeOfCharacterFromSet:
                      [NSCharacterSet controlCharacterSet]];
        
        if (controlRange.location > length)
        {
                return t;
        }
        else
        {
                return [t substringToIndex:controlRange.location];
        }
}

The final piece of logic you need fills in the stub back in method copyStickiesToiPodAtPath:withPrefix:toNotesFolder of AppController. Since this piece of code references methods declared in the Document class, remember to add a #import "Document.h" up at the top of the file.

- (void)copyStickiesToiPodAtPath:(NSString*)path 
                      withPrefix:(NSString*)prefix
                   toNotesFolder:(BOOL)toNotes
{

        //a formatting string
        NSString* STICKIES_VCARD_FORMAT = 
   @"begin:vcard\\nversion:3.0\\nfn:%@\\nnote:%@\\nend:vcard\\n";


        //load the stickies
        NSMutableArray *array = 
                      [[NSMutableArray alloc] initWithArray:
                [NSUnarchiver unarchiveObjectWithFile: 
[@"~/Library/StickiesDatabase" 
                            stringByExpandingTildeInPath]]];

        [array autorelease];
        
        //write each sticky to the path using its title
        NSEnumerator *enumerator = [array objectEnumerator];
        Document *d;
        if (toNotes)
        {
                while (d = [enumerator nextObject]) {
                        NSString *t = 
                        [path stringByAppendingPathComponent:
                                [prefix stringByAppendingString:
                                        [d documentTitleOfLength:15]]];

                        [[d stringValue] writeToFile:t atomically:YES];
                }
        }
        else //to Contacts
        {
                while (d = [enumerator nextObject]) {
                        NSString *t = 
                                [path stringByAppendingPathComponent:
                                        [prefix stringByAppendingString:
                                                [d documentTitleOfLength:12]]];

                        //must replace all newline chars with the 
                        //escaped version for vcard format
                        NSMutableString *escapedStr = 
                                [[NSMutableString alloc] 
                                        initWithString:[d stringValue]];

                        [escapedStr autorelease];
                        [escapedStr replaceOccurrencesOfString:@"\\n" 
                                        withString:@"\\\\n"
                                        options:NSLiteralSearch
                                   range:NSMakeRange(0, [escapedStr length])];
                        
                        [[NSString stringWithFormat:STICKIES_VCARD_FORMAT, 
                                [prefix stringByAppendingString:
                                        [d documentTitleOfLength:12]], 
                           escapedStr, nil] writeToFile:t atomically:YES];
                }
        }
}

You can now "Build and Run" your project and see that your sticky notes synchronize with your iPod when you dock it, or when you select to synchronize from the menu bar.

Final Thoughts

So just how much did you learn in this series? Back in Part 1, we developed a user interface that is "faceless" and lives up in our menu bar, and this turned out to be a nice design decision. The app is always there if we need it, but never in the way if we don't. In this part, we reverse-engineered the StickiesDatabase file so that our own application could read from it and synchronize the notes from the iPod.

Did it seem almost too easy? If you set this article aside, come back once you've forgotten most of what you just read, and try without any help, you might find that it's a bit more time-consuming than it was this time. Unlike "normal" software engineering, reverse-engineering is inherently full of trial-and-error, and requires a great deal of patience and persistence.

Don't do it unless the ends justify the means, you are reasonably certain the internal workings of the software won't change much (if at all), and you're certain there's no better way...or unless it's a rainy day and you feel like having some fun.

StYNCies menu StYNCies preferences Download a more full featured version of the app you just created from russotle.com.

If you'd really like to hone your reverse-engineering skills, try to implement the methods we skipped over that actually modify the StickiesDatabase file. It's not terribly difficult, but requires some insight into "versioning" and working with RTF data a bit more. As promised, the version of the Document class used in StYNCies is available here, but don't peek until you've exhausted your own resources!

Matthew Russell is a computer scientist from middle Tennessee; and serves Digital Reasoning Systems as the Director of Advanced Technology. Hacking and writing are two activities essential to his renaissance man regimen.


Return to MacDevCenter.com.

}

Copyright © 2009 O'Reilly Media, Inc.