Behind-the-Scenes Look at the Judging Process for the Mac OS X Innovators Contest
by Derrick Story
After the close of round one for the Mac OS X Innovators Contest on March 28, I remember scanning through the entries the following day and thinking to myself, "My God, there's a lot of stuff going on here."
When we designed this contest, we thought maybe 10 or 20 people would pull it together and enter the first round. I don't think anyone imagined we'd see the nearly 100 entries logged into our contest site. Beyond that, we're not talking about simple apps either, like the common Euro to U.S. currency exchanger. The list of contestants includes everything from some of the biggest names in Mac developer circles to relative unknowns who have designed beautiful, clever OS X programs.
This great turnout has had an impact on the process, and I want to bring you up to speed on where things stand now, and give you some insight to the judging process.
Notifying Those Who Entered
If you entered the contest before the deadline of March 28, 5 p.m. PST, you should have received a note saying your entry was either accepted and forwarded to the judges, or that it was ineligible. Each of these notes was hand generated from me after reviewing the entry form. To ensure that everyone gets a fair review, there was no automation here. If you entered on time and didn't receive an email from me, please write me immediately at
I did receive some notes of concern after the contest had closed that entrants hadn't heard anything yet about their submissions. Yes, it did take me awhile to review each entry and send the appropriate note. And I apologize for the delay. I'll make some adjustments in the next round (probably along the lines of giving myself more time to notify folks!).
Since I'm also one of the judges, I used this first time through to make my initial notes on the entries themselves. Needless to say, that took a little time, too.
Speaking of Judges
While we're waiting to hear who the winners will be, I want to introduce you to the judges who are working their way through the impressive list of innovative applications submitted. You already know me, but I have lots of help from this crew of bleary-eyed reviewers.
James Duncan Davidson has a nose for clever ideas. And just as important, he understands both Cocoa and Java programming. He's an O'Reilly book author and a perennial speaker at our conferences. Not only can he distinguish how well an application works, he can tell us what's going on beneath the GUI. His initial comment on the field of entries was,"The breadth and depth of the submissions was great to see."
Cory Doctorow has one of the most innovative minds I've met since working at O'Reilly. Not only does he understand the difference between the creative and the mundane, he can pick apart components and figure out how they work. Cory brings a sense of imagination to this judging panel that's essential for a contest that rewards innovation.
Rael Dornfest is the program chair for both the O'Reilly Emerging Technology and the Mac OS X conferences. He's been one of the driving forces behind our wildly successful Hacks series of books, and is a software innovator himself. I just got off the phone with Rael, and the thing that jumped out at me was his sense of what is truly new and fresh, opposed to new wrapping on an old idea. Rael cut through this field of entries like a hot knife through butter. I don't know how he does it.
Daniel Steinberg brings a mathematician's mind to this process. His analytical approach gives me confidence that he won't approve something just because it's "cool." Daniel's background in Java programming also gives him an excellent feel for quality coding. Plus he has an undying sense of fairness (which is an excellent quality in a judge). Daniel's first comment was,"There are a lot of cool ideas coming out of small shops and from single developers."
Nat Torkington is the O'Reilly program chair for the Open Source Convention coming to Portland, Ore., this summer. Nat is a Perl programmer and an extremely well-versed open source guy. He definitely has the big picture when it comes to this family of technologies. I like having Nat on this panel because his finger is on the pulse of innovation in the open source community, and that's an important perspective for a Mac OS X contest.
Together we're pooling our perspectives, and we will come together with an unanimous decision for first and second place.
A Little Bit About the Process
You may be wondering, however, how we go about making those decisions. If you plan on entering the next round, you might want to take a few notes here. Here's the basic approach we've taken to the judging process.
Each judge goes through every entry and reads the abstract and full description. Based on the information in the description, the entry is either moved along to the next round or is eliminated. This highlights the importance of writing informative descriptions.
Next is the testing phase where the surviving software is downloaded and reviewed according to the criteria set forth in the contest rules. Each judge makes his notes and narrows his personal field to five choices in order of preference.
Those selections are sent to me, I add them to my choices, then compile a list of the overall top five candidates based on each judge's ranking and notes. The five finalists are circulated back to each judge for further testing and discussion. Each judge then ranks the five finalists in order of preference and sends them back to me.
I assign a point value to each entry I receive from the judges, based on their personal ranking. I total the points, and the two entries with the highest totals are sent back to the judges for final approval. If a judge doesn't agree with the proposed winners, then we talk it over as a group until we reach consensus.
My overall impression so far has been that the technical caliber of the entries is high, but many lacked the innovation to make it into the final rounds of judging. As we all look to round two, keep in mind that the Mac platform has never been about doing things the same way as everyone else. We are a community of innovators, and our winners should reflect that quality.
Winners will be notified by 5 p.m. PDT on Friday, April 18.
is the author of The Photoshop CS4 Companion for Photographers, The Digital Photography Companion, and Digital Photography Hacks, and coauthor of iPhoto: The Missing Manual, with David Pogue. You can follow him on Twitter or visit www.thedigitalstory.com.
Return to Mac OS X Innovators Contest.
Return to Mac DevCenter.
Comments on this article
1 to 4 of 4
1 to 4 of 4