Monday, February 15, 2010


Should IMAP be replaced?

A friend recently sent me a link to this page, which outlines the author’s idea for a “reimagined Mail Access Protocol” that he would use to replace IMAP. The author starts with the premise that “IMAP is a complex and difficult protocol,” a claim I’ve heard often. My friend says, “I’m not convinced that IMAP is as badly in need of replacement as he says,” and asks what I think.

The short answer is that I’m not convinced either. I also have a much longer answer; read on, if you’re interested in technical details of email protocols.

The author, Gabor, wants his new design to be based on Representational State Transfer (REST) concepts (in the jargon, he’ll make it RESTful). REST architecture is certainly good for many things, and is a useful thing to consider. Unfortunately, as often happens in this business, REST seems to be all the rage these days, the new buzz-word. Everything now has to be RESTful, or it’s not cool enough. I’d want to see a compelling case for throwing away IMAP in favour of a new design, and that would include an analysis of why a new design based on REST would be better than what we have — and would be so much better as to be worth the work, the bugs, the risk, and the deployment problems.

We (the email folks at the IETF) have actually talked about doing a simplification of IMAP and a webmail overlay on it. But there’s that general “IMAP is too complex and difficult” attitude that I really don’t get. Having implemented IMAP on both clients and servers multiple times, I can say that it’s not bad. There’s a little that I would yank out, perhaps... but not a lot. Those who would reject it in favour of “something simpler” would soon find out that there are good reasons for what’s there, and that eliminating something means implementing it in another way. There is, as Heinlein wrote, no such thing as a free lunch.

What’s more, I think the operative bit for Gabor’s outline is here:

One of these days, I’ll write up a spec for the mail access protocol of my dreams. I’m planning to call it the “reimagined Mail Access Protocol” (short: reMAP) which is handy because reMail already owns the domain name

Well, exactly: he needs a business model for his startup. If he can convince enough people that it’s a better mousetrap, he wins.

But is it a better mousetrap? I looked at each of his specific points.

1. All communication over HTTP / HTTPS: TCP connections are great, but for transferring large amounts of email securely, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved. There is no reason to solve them again.

I don’t see how HTTP “solves” any of these things. Mostly, HTTP relies on TCP and TLS/SSL to solve them, all the while he’s saying that “TCP connections” don’t do it. Does he not know that HTTP uses TCP connections, and depends upon TCP features?

2. Stateless: There’s no reason to introduce state like IMAP does with its selected mailboxes. All you need is a HTTP session cookie used for authentication purposes. Session cookie also allow for things like OAuth. OAuth would let third parties get your permission to access your email without having to give them your username and password.

It’s certainly true that you can design a mail access protocol that uses state, or one that doesn’t, and one can argue either way on which is a better design. But if he really thinks that “Session cookie [sic] also allow for things like OAuth,” then he doesn’t understand OAuth. (Apart from that, I have concerns about OAuth — not from a protocol-design point of view, but from a human factors one; maybe that’s another blog post some time.)

3. JSON and UTF-8: All data that’s ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. UTF-8 would be the only encoding allowed, since it is able to represent any character in the Unicode standard.

Who cares about whether what’s on the wire is more human-readable? Honestly.... It’s true that implementors sometimes initially make mistakes in producing or parsing the IMAP protocol. The same is the case for XML and JSON — any format has to be gotten right, but once the code is written and tested, you’re set.

And it’s great to turn everything into UTF-8, but how will he deal with all the difficulties we have today with email and character sets and languages, including stuff that’s mis-tagged or un-tagged? There are far more problems here than can be waved off with a statement that we’ll just use UTF-8 — that’s not a magic wand.

4. Conversations as first-order objects: Gmail, the iPhone SMS app, and Facebook’s messaging system have shown the value of viewing messages not individually but in the context of conversations. In reMAP, the server would be responsible for grouping together messages. While you could still access individual emails, the first-order unit of data would be a conversation.

See the IMAP Thread extension.

5. Labels, not folders: Labels are much for flexible than folders. Each conversation should have multiple labels, and the labels would be included when you request the message, rather than having to scan all folders for the message via IMAP.

An implementation detail. Labels can easily be implemented in IMAP with the Annotate extension. It’s a question of getting the email client software to switch from the folder model to the label model. Of course, you have to get them to do that if you want to replace the protocol, as well.

6. Stable and unique IDs: IMAP has a UID for each message, but it changes the moment you move the message into a different folder. An IMAP server can also declare all UIDs to be invalid at any moment throughout the session. No more! reMAP would have stable and unique IDs for all conversations, emails, and attachments.

Again, Annotate can handle this. Or one can do as many servers do, and use the Message-ID header field.

It might be useful to have an document that specifies some standard annotations that can be used in standard ways, and “labels” and “guid” (globally unique ID) would be good candidates.

7. The beginning of the end for MIME: Yes, you could still get the MIME representation of each message that is sent. But MIME is a messy and complex beast. Instead of requesting the MIME-encoded message parts, you could just ask the server to give you the message as represented in plain text or HTML. Attachments can be downloaded in separate HTTP calls.

Ah, and so he’s attacking MIME as well. It’s true that there are things that should have been done better, and the MIME document editors will be the first to say that, but it’s widely implemented and stable. It’s perfectly fine to turn a received message into HTML when you store it, but, as with item 3, you still have to deal with the MIME that everyone’s sending around, and if you want to replace that you have to figure out how to put those attachments into the message. Unless he’s proposing to replace SMTP as well.

Apart from that, one of the features of IMAP is that it lets the clients not have to deal with MIME, putting the burden on the server (just as he is). There’s really no difference in retrieving multiple message parts through IMAP or through HTTP, is there?

8. Push built in: The two prevailing methods for implementing push email are the IMAP IDLE command (not widely available in IMAP servers) and Microsoft’s ActiveSync, which requires developers to purchase a license from Microsoft. In reMap, clients could just call an HTTP endpoint on the server which returns as soon as new messages are arrive.

IDLE very much is widely available in IMAP servers. Every major one supports it. Even Gmail’s implementation supports it.

And HTTP does not have push either, of course, so he’s proposing using the long-polling hack. The newly chartered HyBi working group should help here.

9. Full-text index on the server: reMAP servers would need to maintain a full-text index of the contents of all messages. There’s no reason clients should be required to download and index everything in order to do an exhaustive full-text search of your email.

And, surprise, IMAP servers do the same thing. Is he unaware of the IMAP SEARCH command?

On the surface, I would say that the guy is either inadequately prepared for this, or is intentionally misrepresenting things in order to promote his idea. It’s certainly possible to design a slimmer, groovier email access protocol — no one, not even the original IMAP author, thinks IMAP is the Best of All Possible Worldstm — but this doesn’t seem to be much of a step toward that, and it’s not clear how much it’s needed anyway. The budding effort in the IETF died for lack of real interest.


Ray said...

Barry, Barry... you just don't get it, do you? This must be what everyone needs, since it has a much catchier acronym.

Sue VanHattum said...

I read this, even though I didn't understand most of it. As a user, I disagree with #5. I like folders better than labels.

The Ridger, FCD said...

The very least I demand from an "upgrade" (not that my demands are met, of course) is that it not remove any capability I currently have. After that, I want it to be much better than what I have in direct proportion to how hard it will be for me to learn it, and how much down time there will be associated with getting it. If all you have is a spiffy new way to do what I already do - then, honestly, why?

HRH said...

I think, as you pointed out, he is not adequately prepared for this, however if at some point, the working group at IETF, decides to implement, a more suitable messaging protocol, the flexibility factor ought to be one of the focal points. For instance, my friend just switched to Gmail, last month, she utterly hates the conversation feature in Gmail, and finds it very confusing, and I on the other hand, love this feature. My brother doesn’t care about the “label” feature in Gmail, I again, love it (as you might have guessed, I am an avid Gmail subscriber), thus a flexibility in a protocol would be essential to cater to different users. I just used these two examples to drive a point about flexibility. For example it would be great, if one user has the ability to create “folder” to categorize her email, while another user has the option to use “label” to accomplish that, on the same server. Of course one might argue that presenting mail categories, in a “folder” or “label” form is a matter of implementation and that might be true but not all issues, would fall under implementation, and need to be addressed at the protocol level.

Tony Romano

Barry Leiba said...

Flexibility, yes, I agree. But be careful not to confuse what happens in the protocol with what happens in the mail client. A protocol needs to be well defined and carefully designed, and, often, "flexibility" is a disaster in protocols.

A protocol presents an abstraction, and what the protocol needs are elements that allow implementations to be flexible in how they present things to their users.

Gmail, with respect to IMAP, is an excellent example. Internally, Gmail uses labels. But it presents them as folders to IMAP users, and from the IMAP side you can't tell the difference. But to properly use labels in a client, this doesn't work — a client can select three folders and find the same message in each, but it can't look at a message and say "tell me all the folders it's in," because that's not the abstraction presented by IMAP.

In other words, if IMAP abstracted labels instead of folders, clients could implement folders on top of that. But that's not true as IMAP was defined. (Of course, as I said, the Annotate extension provides a good way to do labels.)

I actually use Gmail in both ways, to take advantage of different strengths. When I want the "conversation" view (which is actually most of the time now), I use the webmail client. When I don't, I use an IMAP client to look at the "folder" (label) in question.

Sue VanHattum said...

I don't use gmail's labels because I can usually find what I want by searching, and that seems easier. (I think I got a bit stuck on the conceptual shift from folders to labels.)

I think gmail has a bug in its conversation feature. I think when I get two new emails in the same thread, the inbox count doesn't go up properly, and I skip over a message without noticing. I'm not exactly sure of all the details, but it has happened multiple times.

My other problem is finding a particular reply in a long conversation.

But it's still way nicer than hotmail. (Sorry, Barry, I know this is totally off the topic of IMAP.)

HRH said...

Sue said..
>I think gmail has a bug in its conversation feature. I think when I get two new >emails in the same thread, the inbox count doesn't go up properly, and I skip >over a message without noticing. I'm not exactly sure of all the details, but it has >happened multiple times.

I have been a Gmail subscriber since 2005, and haven’t experienced this bug yet; however this doesn’t imply that there isn’t one; it may not be a persistent bug, and rather an intermittent one. Having said that, I would like to point out, something about the message counter in Gmail, which might be of an interest to you:

Gmail, maintains, 2 counters to display two different information about the incoming messages to the user (I actually, prefer to avoid using the word “message”, in order to be inline, with Gmail semantics, thus I will use the word conversation instead). The first counter is at the inbox level, indicating the number of “unread” conversations in the inbox. The second counter is at the conversation level, indicating, the total of replies within a given conversation. If the number of replies increases, within a given conversation then the inbox counter, does not increment and only the conversation counter does. Aside from this, there is another interesting difference between these two counters, as they serve different objectives. As user begins to read her new conversation(s), the inbox counter will decrement but this is not true about the conversation counters. In the other word, as she reads the individual replies within a given conversation, the conversation counter remains unchanged. To elaborate this better with an example, let’s assume, you have (10) unread conversations in your inbox. You decide to send an email, to Barry and me about a dinner invitation for Tuesday evening at 7:30 pm requesting RSVP. Barry emails you back and accepts the invitation. You access your Gmail inbox and notice the inbox counter is at (11) showing a new conversation in your inbox, at the same time the conversation counter on the newly received email from him is at (2). Moments later I respond to your email and you will notice the inbox counter is still at (11), and the conversation counter is at (3). This really indicates that my reply hasn’t added any new conversation to the inbox and rather to the conversation itself. You open the conversation and read our replies then you notice the inbox counter is back to (10) and the conversation counter is still at (3). This feature of Gmail is very different than the traditional messaging clients like, Hotmail, Yahoo, Excite and so on which causes confusion among some newly Gmail subscriber.

> My other problem is finding a particular reply in a long conversation.

Although Gmail uses an excellent search engine, this can happen when using any messaging client and not limited to Gmail.

> I don't use gmail's labels because I can usually find what I want by searching, and that seems easier. (I think I got a bit stuck on the conceptual shift from folders to labels.)

I agree with you, it is a matter of conceptual shift. Even though I love “label”, I think it falls short to the traditional “folder” style in one major aspect, and that’s the ability to create “sub-label”, analogues to “sub-folders”.

I would like to thank Barry for allowing me to make this post, since this is off the original topic of “IMAP .vs. REMAP” (LOL, I just find the term REMAP amusing).

If you need to discuss the Gmail topics further, just ask Barry for my email and we do it offline.

Sue VanHattum said...

Ahh, got it. No bug in their opinion then. I've just experimented, and maybe I won't miss messages now. (I'll always read until I see the reply box, instead of stopping at the person's closing.) Thanks for the explanation!