MailNews:Better Faster IMAP Plan

Leverage Offline capabilities to make online more responsive

The basic idea is to use the offline capabilities of Thunderbird while online to make the UI more responsive, when operating on IMAP messages, out of the box, without the user having to tweak any settings. (Note: this applies to SMTP sending as well)

UX Decisions to make

With an assumption that all operations are to be perceived as more responsive, many of our decisions are made for us. A proxy increases speed also increase the need for good feedback because operations appear to have completed quickly and yet may not have actually taken place.

• are all operations to be available offline?

• how do we transition existing users to the new model without causing pain?

• The offline model will make "Unsent Items" more important -- should we promote it to a high-visibility UI element (like Mail.app's Outbox)?
  • Until we have a failure in sending the Outbox doesn't offer much value, however once a failure occurs it needs to be high-visibility (clarkbw)
  • Do you suggest a virtual Outbox that will show up only there are pending messages? (emre)

• What UI do we present to allow users to override the auto-download of all message bodies? Is it per folder? Per account? Global?
  • This problem needs to be broken out into what I believe are 2 separate concerns. (clarkbw)
    • Concern 1: Change. Some people will be concerned about this change no matter what and will want an "off switch" or will complain loudly.
    • Concern 2: Space. Other are probably concerned with space issues of auto-downloading, this requires some thought and is most likely the real underlying concern of #1.

• Do we give the user feedback when the delete actually happens? (davida: IMO no, only when delete fails)
  • We need to give feedback for every operation but on different levels (clarkbw)
    • Notice feedback for operations that are completed
    • Error feedback (and rollback) for operations that fail

• Do we make the Trash an offline folder like all other folders? (davida: Why not?)
  • We need to make Trash fast, responsive, and work offline (clarkbw)

• Are there other imap operations that we would want to do "offline", like renaming a folder, or should we stick with the most often used commands, like reading a message, move, delete, SMTP send, etc?
  • Why not? (clarkbw)
  • Draft Messages

Next step is to look at this from some experiences we want to happen, sending mail, receiving mail, deleting mail.

Development Strategy

1. Consider implementing what follows using a "proxy" to mediate between the UI and the IMAP protocol layer (to be reusable w/ other protocols)
2. Switch pref defaults
   • switch the default for imap folders so that they're configured for offline use out of the box. (code URL?)
   • switch mail.server.default.autosync_offline_stores to true, by default
3. Do UI operations in offline mode always
   • change imap move/deletes though the UI so that they're done offline, and the offline operation is played back after the next message is displayed, or immediately if there's no next message to display.
4. Come up with a message header and message body download strategy
   • We probably want to download recent information before older information. The most succinct form of the command to fetch multiple messages (bodies or headers) fetches them in UID, roughly chronological order, using ranges, e.g., 1-20. But you certainly could fetch 20,19,18,17 etc. Are you talking about the new profile case, or the every day case? For the new profile case, a simple thing to do would be to fetch the bodies of the unread messages first, then the bodies of the read messages. Or we could fetch ranges of messages, but start at the end, not the top, e.g., 80-100, 60-79, etc.
   • We should be careful about downloading automatically large message bodies, as they may block IMAP calls.
   • This would allow us to consider applying the same sort of logic to other protocols (NNTP?)
5. Come up with an offline operation playback strategy
   • May be worth implementing a few and seeing which ones work out best in practice.
6. Address consequences of preference changes
   • address growth of offline store folder - come up with a compacting strategy that is user-efficient. (once a week? once a month? Once a file reaches some % growth since the last compacting?)
   • Make sure that offline operations that fail give the user appropriate context as to the failure (offer to open up original message when sends fail, etc.)
7. NB: At this point, the system should be releasable
8. Consider further enhancements
   • Use the new platform code to detect whether there is a net connection or not, and 1) display that status to the user somehow, and 2) use the status to determine when to replay offline operations.
   • Use the Idle service to download message bodies when the user is idle, or to compact offline stores
   • Breakup the download of message bodies into multiple passes, which would allow the user to sneak in and start reading messages before we've synced the whole folder.
   • Playback offline operations when the user is idle, though I think we'd run into less issues if we tried to do the offline playback closer to the UI event.

Emre: Proposal for Implementation

I like to break down these requirements into 3 distinct implementation steps:

• Offline operation playback feature implementation
• Preemptive/Automatic message download feature implementation
• Background message send feature implementation

1. Offline operation playback feature: Delete operation covers two different operations at the same time: Delete and Move. These two server-side operations cover the majority of use cases that require better offline support. Makes sense to me to start implementing "Proxy" focused on Delete operation initially, since it provides good granularity for deployment and testing.
   1. Implement offline support for "Delete" operation:
      • Deleting messages, considering two flavors of delete;
        • Move to Trash
        • Delete immediately
      • Deleting folders
   2. Implement offline support for other operations (rename, flags, copy, create)
2. Preemptive/Automatic message download feature:
   1. Modify current offline features
      • Switch the default for imap folders so that they're configured for offline use out of the box
      • Switch mail.server.default.autosync_offline_stores to true, by default
   2. Implement automatic compacting mechanism for mbox files
   3. Re-factor current offline mechanism to implement strategy-based message body download prioritization.
3. Background message send feature:
   1. Implement a "Unsent" folder UI element for IMAP folders
   2. Modify existing code to:
      • Create a new nsMsgFolder type
      • Show pending messages inside this folder - allow limited user interaction (cancel only?)
      • Don't show "sending" dialog anymore