Should IMAP be replaced?

A friend recently sent me a link to this page, which outlines the author’s idea for a “reimagined Mail Access Protocol” that he would use to replace IMAP. The author starts with the premise that “IMAP is a complex and difficult protocol,” a claim I’ve heard often. My friend says, “I’m not convinced that IMAP is as badly in need of replacement as he says,” and asks what I think.

The short answer is that I’m not convinced either. I also have a much longer answer; read on, if you’re interested in technical details of email protocols.

The author, Gabor, wants his new design to be based on Representational State Transfer (REST) concepts (in the jargon, he’ll make it RESTful). REST architecture is certainly good for many things, and is a useful thing to consider. Unfortunately, as often happens in this business, REST seems to be all the rage these days, the new buzz-word. Everything now has to be RESTful, or it’s not cool enough. I’d want to see a compelling case for throwing away IMAP in favour of a new design, and that would include an analysis of why a new design based on REST would be better than what we have — and would be so much better as to be worth the work, the bugs, the risk, and the deployment problems.

We (the email folks at the IETF) have actually talked about doing a simplification of IMAP and a webmail overlay on it. But there’s that general “IMAP is too complex and difficult” attitude that I really don’t get. Having implemented IMAP on both clients and servers multiple times, I can say that it’s not bad. There’s a little that I would yank out, perhaps... but not a lot. Those who would reject it in favour of “something simpler” would soon find out that there are good reasons for what’s there, and that eliminating something means implementing it in another way. There is, as Heinlein wrote, no such thing as a free lunch.

What’s more, I think the operative bit for Gabor’s outline is here:

One of these days, I’ll write up a spec for the mail access protocol of my dreams. I’m planning to call it the “reimagined Mail Access Protocol” (short: reMAP) which is handy because reMail already owns the domain name

Well, exactly: he needs a business model for his startup. If he can convince enough people that it’s a better mousetrap, he wins.

But is it a better mousetrap? I looked at each of his specific points.

1. All communication over HTTP / HTTPS: TCP connections are great, but for transferring large amounts of email securely, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved. There is no reason to solve them again.

I don’t see how HTTP “solves” any of these things. Mostly, HTTP relies on TCP and TLS/SSL to solve them, all the while he’s saying that “TCP connections” don’t do it. Does he not know that HTTP uses TCP connections, and depends upon TCP features?

2. Stateless: There’s no reason to introduce state like IMAP does with its selected mailboxes. All you need is a HTTP session cookie used for authentication purposes. Session cookie also allow for things like OAuth. OAuth would let third parties get your permission to access your email without having to give them your username and password.

It’s certainly true that you can design a mail access protocol that uses state, or one that doesn’t, and one can argue either way on which is a better design. But if he really thinks that “Session cookie [sic] also allow for things like OAuth,” then he doesn’t understand OAuth. (Apart from that, I have concerns about OAuth — not from a protocol-design point of view, but from a human factors one; maybe that’s another blog post some time.)

3. JSON and UTF-8: All data that’s ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. UTF-8 would be the only encoding allowed, since it is able to represent any character in the Unicode standard.

Who cares about whether what’s on the wire is more human-readable? Honestly.... It’s true that implementors sometimes initially make mistakes in producing or parsing the IMAP protocol. The same is the case for XML and JSON — any format has to be gotten right, but once the code is written and tested, you’re set.

And it’s great to turn everything into UTF-8, but how will he deal with all the difficulties we have today with email and character sets and languages, including stuff that’s mis-tagged or un-tagged? There are far more problems here than can be waved off with a statement that we’ll just use UTF-8 — that’s not a magic wand.

4. Conversations as first-order objects: Gmail, the iPhone SMS app, and Facebook’s messaging system have shown the value of viewing messages not individually but in the context of conversations. In reMAP, the server would be responsible for grouping together messages. While you could still access individual emails, the first-order unit of data would be a conversation.

See the IMAP Thread extension.

5. Labels, not folders: Labels are much for flexible than folders. Each conversation should have multiple labels, and the labels would be included when you request the message, rather than having to scan all folders for the message via IMAP.

An implementation detail. Labels can easily be implemented in IMAP with the Annotate extension. It’s a question of getting the email client software to switch from the folder model to the label model. Of course, you have to get them to do that if you want to replace the protocol, as well.

6. Stable and unique IDs: IMAP has a UID for each message, but it changes the moment you move the message into a different folder. An IMAP server can also declare all UIDs to be invalid at any moment throughout the session. No more! reMAP would have stable and unique IDs for all conversations, emails, and attachments.

Again, Annotate can handle this. Or one can do as many servers do, and use the Message-ID header field.

It might be useful to have an document that specifies some standard annotations that can be used in standard ways, and “labels” and “guid” (globally unique ID) would be good candidates.

7. The beginning of the end for MIME: Yes, you could still get the MIME representation of each message that is sent. But MIME is a messy and complex beast. Instead of requesting the MIME-encoded message parts, you could just ask the server to give you the message as represented in plain text or HTML. Attachments can be downloaded in separate HTTP calls.

Ah, and so he’s attacking MIME as well. It’s true that there are things that should have been done better, and the MIME document editors will be the first to say that, but it’s widely implemented and stable. It’s perfectly fine to turn a received message into HTML when you store it, but, as with item 3, you still have to deal with the MIME that everyone’s sending around, and if you want to replace that you have to figure out how to put those attachments into the message. Unless he’s proposing to replace SMTP as well.

Apart from that, one of the features of IMAP is that it lets the clients not have to deal with MIME, putting the burden on the server (just as he is). There’s really no difference in retrieving multiple message parts through IMAP or through HTTP, is there?

8. Push built in: The two prevailing methods for implementing push email are the IMAP IDLE command (not widely available in IMAP servers) and Microsoft’s ActiveSync, which requires developers to purchase a license from Microsoft. In reMap, clients could just call an HTTP endpoint on the server which returns as soon as new messages are arrive.

IDLE very much is widely available in IMAP servers. Every major one supports it. Even Gmail’s implementation supports it.

And HTTP does not have push either, of course, so he’s proposing using the long-polling hack. The newly chartered HyBi working group should help here.

9. Full-text index on the server: reMAP servers would need to maintain a full-text index of the contents of all messages. There’s no reason clients should be required to download and index everything in order to do an exhaustive full-text search of your email.

And, surprise, IMAP servers do the same thing. Is he unaware of the IMAP SEARCH command?

On the surface, I would say that the guy is either inadequately prepared for this, or is intentionally misrepresenting things in order to promote his idea. It’s certainly possible to design a slimmer, groovier email access protocol — no one, not even the original IMAP author, thinks IMAP is the Best of All Possible Worlds^tm — but this doesn’t seem to be much of a step toward that, and it’s not clear how much it’s needed anyway. The budding effort in the IETF died for lack of real interest.