MailNews:Better Faster IMAP Plan

Leverage Offline capabilities to make online more responsive

The basic idea is to use the offline capabilities of Thunderbird while online to make the UI more responsive, when operating on IMAP messages, out of the box, without the user having to tweak any settings. (Note: this applies to SMTP sending as well)

UX Decisions to make

With an assumption that all operations are to be perceived as more responsive, many of our decisions are made for us. A proxy increases speed also increase the need for good feedback because operations appear to have completed quickly and yet may not have actually taken place.

• are all operations to be available offline?

• how do we transition existing users to the new model without causing pain?

• The offline model will make "Unsent Items" more important -- should we promote it to a high-visibility UI element (like Mail.app's Outbox)?
  • Until we have a failure in sending the Outbox doesn't offer much value, however once a failure occurs it needs to be high-visibility (clarkbw)
  • Do you suggest a virtual Outbox that will show up only there are pending messages? (emre)

• What UI do we present to allow users to override the auto-download of all message bodies? Is it per folder? Per account? Global?
  • This problem needs to be broken out into what I believe are 2 separate concerns. (clarkbw)
    • Concern 1: Change. Some people will be concerned about this change no matter what and will want an "off switch" or will complain loudly.
    • Concern 2: Space. Other are probably concerned with space issues of auto-downloading, this requires some thought and is most likely the real underlying concern of #1.

• Do we give the user feedback when the delete actually happens? (davida: IMO no, only when delete fails)
  • We need to give feedback for every operation but on different levels (clarkbw)
    • Notice feedback for operations that are completed
    • Error feedback (and rollback) for operations that fail

• Do we make the Trash an offline folder like all other folders? (davida: Why not?)
  • We need to make Trash fast, responsive, and work offline (clarkbw)

• Are there other imap operations that we would want to do "offline", like renaming a folder, or should we stick with the most often used commands, like reading a message, move, delete, SMTP send, etc?
  • Why not? (clarkbw)
  • Draft Messages

Next step is to look at this from some experiences we want to happen, sending mail, receiving mail, deleting mail.

Development Strategy

1. Consider implementing what follows using a "proxy" to mediate between the UI and the IMAP protocol layer (to be reusable w/ other protocols)
2. Switch pref defaults
   • switch the default for imap folders so that they're configured for offline use out of the box. (code URL?)
   • switch mail.server.default.autosync_offline_stores to true, by default
3. Do UI operations in offline mode always
   • change imap move/deletes though the UI so that they're done offline, and the offline operation is played back after the next message is displayed, or immediately if there's no next message to display.
4. Come up with a message header and message body download strategy
   • We probably want to download recent information before older information. The most succinct form of the command to fetch multiple messages (bodies or headers) fetches them in UID, roughly chronological order, using ranges, e.g., 1-20. But you certainly could fetch 20,19,18,17 etc. Are you talking about the new profile case, or the every day case? For the new profile case, a simple thing to do would be to fetch the bodies of the unread messages first, then the bodies of the read messages. Or we could fetch ranges of messages, but start at the end, not the top, e.g., 80-100, 60-79, etc.
   • We should be careful about downloading automatically large message bodies, as they may block IMAP calls.
   • This would allow us to consider applying the same sort of logic to other protocols (NNTP?)
5. Come up with an offline operation playback strategy
   • May be worth implementing a few and seeing which ones work out best in practice.
6. Address consequences of preference changes
   • address growth of offline store folder - come up with a compacting strategy that is user-efficient. (once a week? once a month? Once a file reaches some % growth since the last compacting?)
   • Make sure that offline operations that fail give the user appropriate context as to the failure (offer to open up original message when sends fail, etc.)
7. NB: At this point, the system should be releasable
8. Consider further enhancements
   • Use the new platform code to detect whether there is a net connection or not, and 1) display that status to the user somehow, and 2) use the status to determine when to replay offline operations.
   • Use the Idle service to download message bodies when the user is idle, or to compact offline stores
   • Breakup the download of message bodies into multiple passes, which would allow the user to sneak in and start reading messages before we've synced the whole folder.
   • Playback offline operations when the user is idle, though I think we'd run into less issues if we tried to do the offline playback closer to the UI event.

Emre: Proposal for Implementation

I like to break down these requirements into 3 distinct implementation tasks:

• Offline operation playback feature implementation (needs a better name)
• Preemptive/Automatic message download feature implementation
• Background message send feature implementation

1. Offline operation playback feature Delete operation covers two different operations at the same time: Delete and Move. These two server-side operations cover the majority of use cases that require better offline support. Makes sense to me to start implementing "Proxy" focusing on Delete operation initially because it provides good granularity for deployment and testing. Possible milestones are;
   1. Implement offline support for "Delete" operation:
      • Deleting messages, considering two flavors of delete;
        • Move to Trash
        • Delete immediately
      • Deleting folders
   2. Implement UI feedback mechanism (UI elements and logic)
   3. Implement offline support for other operations (rename, flags, copy, folder creation)
2. Preemptive/Automatic message download feature Possible milestones are:
   1. Modify current offline features
      • Switch the default for imap folders so that they're configured for offline use out of the box
      • Switch mail.server.default.autosync_offline_stores to true, by default
   2. Implement automatic compacting mechanism for mbox files
   3. Re-factor current offline mechanism to implement strategy-based, automatic message body/attachment download. Couple ideas are:
      • Prioritize by date
      • Prioritize by sender
      • Prioritize by size, or attachment count
      • Idle service usage
      • Multi pass download for messages with multiple attachments
   4. Implement UI elements and logic
3. Background message send feature Possible milestones are:
   1. Implement a "Unsent" folder UI element for IMAP folders
   2. Modify existing code to:
      • Create a new nsMsgFolder type
      • Show pending messages inside this folder - allow limited user interaction (cancel only?)
      • Don't show "sending" dialog anymore

Task List for Feature 1

1. Implement Proxy mechanism (Estimation: ?)
   • Shall map local folder model to server folder model, vice versa.
   • Shall execute commands on background - decouple folder event from protocol command.
   • Shall notify UI if the operation is not successful
   • (?) Shall store commands persistently in case that TB exits while there are pending operations in the Q
   • Shall deal with unsolicited UI events and IMAP server messages.
   • Shall rollback the UI operation if the command is not successful.
   • Shall be smart enough to deal with event chaining, for example; Assume that Delete mode is "MoveToTrash", the user deletes 100 messages. TB moves the messages into the trash immediately and starts the operation on the background. Then the user moves 50th deleted message back into the INBOX and mark it as "Not Read" before the "move to trash" command is sent to the server. Proxy should be smart enough not to lose the message identifier, and change the command chain if necessary - moving a message into trash might change its identifier on the server, so the content of the pending command.
2. Re-factore Imap protocol layer to handle error cases gracefully, such as no-connection, server error, command execution error. Note: Put current threading model into consideration. When the user exceeds the number of cached-connections, all pending operations waiting for this session should be executed before recycling the thread. Thread recycling causes disconnection form the server which might cause an implicit EXPUNGE command on the server. (Estimation: ?)
3. Implement UI feedback mechanism (Estimation: ?)

Total: ?

Risks: [TBD]

Decisions to make

1) Should we warn the user about pending offline operations before exit?

• if yes;
  • should we give the user an option to store them locally for the next time?
  • or should we do it automatically
• if no;
  • What's the right action to take?

Even this is a rare case, it is inevitable: imagine a scenario where the user deletes 200 messages and tries to exit immediately.

>We should warn the user about pending operations and give the option to either execute them before exit TB, or save them for the next time.[Emre]

2) Offline operation storage:

• Is offline operation persistency required?
  • If yes, for which operations? Delete? Rename? Flag change? Tagging?
• How long should we store it?
• How to deal with back-dated unexecuted offline operations?
• What kind of store we should use? Mork, Sqlite, text file
• Is security a concern - encryption?

3) UI feedback for:

• Operations that are completed
• Error feedback (and rollback) for operations that fail
• Feedback for partially or not downloaded messages (?)

4) When do we playback offline operations: > We can deal with this problem by making a design that is flexible enough to implement different strategies. Don't believe it would cost us much in term of time. [Emre]

Design Proposal

See current design [TBD]

Task List for Feature 2

[TBD]

Decisions to make

[TBD]

Task List for Feature 3

[TBD]

Decisions to make

[TBD]