MacDevCenter    
 Published on MacDevCenter (http://www.macdevcenter.com/)
 See this if you're having trouble printing code examples


Email Protocols: Where Do We Go Now?

by Giles Turnbull
08/29/2006

Ah, email.

Everyone has an opinion about it. Some of us complain about getting too much of it. Many people find it a drain on their time, productivity, and creativity. For most, spam is such a problem that it effectively renders the whole concept of email broken.

But despite all of the things that are wrong, or can go wrong with email, most of us continue to use it, day after day.

Despite being an old technology, email still has room to evolve. Despite all of its problems, we show no signs of giving it up, so it makes sense to keep an eye on the evolutionary changes and make the best of a bad situation.

What follows is a brief guide to the various flavors of email protocol; it's not intended as a technical paper for those who already speak fluent POP, but rather as an explanatory review for the rest of us, with pointers to possible evolutionary steps to come.

Traditional Email Basics

At the most basic level, POP and IMAP are simply cousins with different ways of doing the same thing. Think of your email account as an in-tray on a server somewhere, where all of your incoming email arrives and waits until you come along to read it. Using POP, you empty the contents of the in-tray and download all of the messages inside it to your local disk. They are moved from one location to another.

Using IMAP, your email client becomes a tool for manipulating the messages as they are stored on the remote computer. Nothing, except for client preferences, is kept on your local disk. The messages are stored remotely and can be accessed using pretty much any computer.

POP is simpler to implement and easier for ordinary people to understand, but it isn't very flexible. When the time comes to switch from one computer to another, or from an old client to a new one, you face the headache of transferring or copying all of the messages and their associated metadata. Data loss is not uncommon here.

The good thing about IMAP is that your choice of client or computer really doesn't matter. You can flip between multiple computers, clients, and locations without worry. As long as you know your IMAP settings, you can reach your email from anywhere. The bad thing is that IMAP is a great deal more complicated to set up, and it can be confusing for users, especially newbies, to understand that their email is "somewhere else" and not on the computer in front of them.

IMAP and POP In More Detail

POP, or Post Office Protocol, or POP3, was designed with dial-up users in mind. Hardly surprising given that, at the time (the mid-1980s), high-speed internet access was almost nonexistent.

POP typically allows the client access to a single mailbox on the remote server. From this server, it can download any new or saved messages to the client computer's hard disk, at which point the messages are (normally) deleted from the server.

IMAP, or Internet Message Access Protocol, is "a protocol which enables an advanced distributed client/server electronic mail paradigm" in the words of its inventor, Mark Crispin of the University of Washington. Mr. Crispin created IMAP in 1985 and continues to work on it and the IMAP Toolkit to this day.

IMAP is now at version 4 revision 1, which itself is older than you might expect, published over three years ago in 2003.

The whole point of IMAP is that your email is stored remotely. It's a classic use of the "client/server" model, with your email on the server, and you having access to one or more client applications with which you read and manage messages. It overcomes many of the restrictions and difficulties people used to encounter when using POP, especially professional computer users who have traditionally been more likely to own or use more than one machine.

In addition, it supports several modes of operation, overcoming the obvious drawback of the client/server approach: the need for a network connection between the two. In "offline" mode, IMAP enables the user to continue working on new draft messages and stored copies of old incoming mail, and then synchronizes everything as necessary when a network connection is available again.

IMAP brings extra features that go some way beyond what POP could ever offer. It allows the client access to multiple folders, and indeed the right to manipulate folders directly on the server. There's also the option to share folders with other users; with this in place, people can collaborate on work without the need to endlessly copy or forward messages among one another. IMAP offers users the chance to create their own flexible folder hierarchy. It's a far more powerful protocol than POP, although that makes it somewhat more complicated to use.

That might explain why, historically, it has lagged behind POP in terms of numbers of users. Certainly during the 1990s consumer internet boom, the vast majority of internet service providers (ISPs) only thought to offer POP accounts to their users, spurning the more complex and server processor-hungry IMAP option.

But that situation is changing, not least because having more than one computer is becoming increasingly common outside computer-professional circles. How many people do you know with both a desktop and a notebook computer? How many households where all family members have their own machines? With computer prices diving ever lower, mass consumer use of multiple computers is becoming more common, so demand for IMAP (or IMAP-like) email services is rising.

In recent years, ISPs have started to change their ways. It's more common now to find IMAP offered alongside POP, although POP is often the default option.

Ultimately, IMAP is a more flexible and powerful means of managing email, and its popularity is growing. Back in 1998, IMAP4 developer John Myers predicted, "I think IMAP will eventually replace POP," although that might be taking longer than he anticipated. At the time of writing, Google Trends suggests a gradual declining interest in POP3 but not much growing interest in IMAP4.

Google Trends
Figure 1. Google Trends

IMAP Clients for Mac OS X

Mac OS X is not short on clients that will happily talk to IMAP servers. The most obvious choice is Apple's own Mail application, which supports both without any fuss. It is not the speediest of IMAP clients, however, and the internet is stuffed full of annoyed Mail users publicly calling on Apple's engineers to make it faster. (Personally speaking, Mail's slowness with IMAP led me to abandon it after a few months of use.)

Two perfectly decent alternative GUI IMAP clients are Thunderbird and Eudora. Thunderbird earns extra positive karma thanks to being free, and while Eudora's somewhat outdated looks might put you off, its incredible speed and stability are well worth it. PowerMail and Entourage are also worth considering.

Mulberry has been around for a long time, and is a full-fledged (and now free) IMAP client. Despite many glowing reviews, it never really gained the large customer base it needed and sadly, last year, Mulberry's parent company went under. On the positive side, this brought about the re-release of the app as freeware, and, rumor has it, the hiring of Mulberry's creator Cyrus Daboo (described by company co-founder Matt Wall as "incredibly smart and dedicated") by Apple. Perhaps he's already begun working some of his magic there.

Of course, if you prefer your software without a GUI, there's always Mutt or Pine; Fink is there to make things easier for people lacking Unix-fu.

And Whatever The Heck Gmail Is

Gmail enjoys an excellent reputation right now, attracting lots of users--many of them serious geeks. Why? Because it offers excellent features, is fast and dependable, and free and flexible (it'll even let you use other email addresses, or handle email for an entire domain for you). It's not IMAP and it's not POP; it's webmail that Just Works, and that means a simpler experience for users and fewer headaches for administrators.

Thumbnail, click for full-size image.
Figure 2. Gmail Settings (Click for full-size image)

Is IMAP really going to replace POP? Or will webmail, in the form of Gmail and its inevitable competitors, render both of them redundant in the long term?

As we have seen in the last few days with the release of Google Apps for Your Domain, Gmail is just one aspect of a wider strategy on Google's part: an attempt to bring essential office or groupware applications to users through a browser. Gmail itself began life as an internal project, used by Google employees. In March 2004, an official beta testing phase began and a handful of outsiders were invited to join in. On April Fools Day that year, an official launch announcement was made--much to the amusement of almost everyone, since it was widely considered a joke.

That, in itself, demonstrates how innovative Gmail's approach was. So different was its design to traditional web-based email services, people were more inclined to think it was a practical joke than a real service.

And because it is webmail, Gmail neatly sidesteps any questions about email protocols. There's no need for users to consider whether they need a POP or an IMAP client; all they need is a browser (or a phone, or a PDA). Maybe even just a Chumby.

Slight Aside: Text Messaging?

Email persists as the primary means of communication among internet users, at least those of a certain age. That said, there are clear signs that the younger generation is more interested in text messaging and Instant Messages than in email.

Text messaging is faster, simpler, and more instant than email. An email inbox tends to be left until its owner is ready to go through the messages and deal with them; an incoming text message tends to be answered sooner, if not immediately. There's a "conversational" feel to texting that email lacks.

This quote from Paul Golding is telling: "Email is like placing a letter in someone's in-tray, whereas texting is like tapping them on the shoulder and saying look at this, whilst placing a message on a slip of paper in their hand."

Text messages have become hugely popular, especially in Europe, but they were not originally created for consumer use. The idea was to use them as a means of sending network and account information to users, not as user-to-user communication. The instantaneous results and low cost (in comparison to voice calls) were what made them very popular. As phones themselves get more powerful and technically complex, text messaging may be replaced by other technologies--perhaps some development of Instant Messaging. But I'd be willing to bet it will be known as texting for some time afterwards, especially here in the U.K. where it has become embedded in youth culture.

But I digress... let's get back to email and take stock of our situation.

Where Do We Go Now?

Ah, email.

Having established that email has a complicated history, tends to drive people crazy, has had entire books written about keeping it under control, and is broken beyond all repair for some people, let's try to work out what we can do--practically speaking--to make using it less of a hassle.

For casual and lightweight email users, there's nothing at all wrong with a good old-fashioned POP account and a copy of Apple's Mail. Assuming the ISP at the other end can be relied on, Mail copes very well with the basics and has the intuitive kind of interface that casual users can get along with (which isn't to say that pro users can't get a lot from it, too). Webmail is still an option here, of course. We know that casual users like webmail--look at the huge international success of Hotmail.

For professionals who live half their working day in email, combining IMAP and Gmail makes sense, but only if they (and their employers) are prepared to place sufficient trust in Google to keep business secrets secret.

In practical terms this makes a lot of sense, since using Gmail while travelling can help you get around SMTP problems (where your normal SMTP server might only permit you access if you're connecting via a certain ISP or IP range). Combining the two means Gmail effectively becomes an online backup of every email message in your IMAP account--a free, huge, and easily searched backup.

This combination is clearly popular with geeks, too. In my personal experience, and despite privacy concerns from many people, a lot of geeks I know have switched fully or partially to Gmail from existing IMAP setups. Some just use Gmail as a backup, or depend on it as a spam filter or for managing mailing lists; others depend on it completely for all their daily email tasks. There are, of course, those who say the privacy concerns are nothing to worry about; at least, they are no more of a concern than almost any other email service. People have grown used to the idea that email is, in general, a secure and private means of communication, which it is far from being (unless you take a proactive step and use encryption).

For those who are uncomfortable about the privacy issues, there are alternatives. Fastmail is a popular choice, offering very flexible IMAP accounts with webmail for a reasonable annual fee. Hushmail is an attractive service, built entirely around the premise of security for your email.

Final Thoughts

POP? Gmail? IMAP?

Not everyone thinks very much of email in general. And, not everyone likes IMAP.

And yet email persists, it grows, and people are always interested in new devices that support it. As mobile devices evolve, our messaging services will, too. I think we're going to see a lot of change in the next few years.

Giles Turnbull is a freelance writer and editor. He has been writing on and about the Internet since 1997. He has a web site at http://gilest.org.


Return to the Mac DevCenter.

Copyright © 2009 O'Reilly Media, Inc.