MailNews:Better Faster IMAP Plan

Leverage Offline capabilities to make online more responsive

The basic idea is to use the offline capabilities of Thunderbird while online to make the UI more responsive, when operating on IMAP messages, out of the box, without the user having to tweak any settings. (Note: this applies to SMTP sending as well)

UX Decisions to make

With an assumption that all operations are to be perceived as more responsive, many of our decisions are made for us. A proxy increases speed also increase the need for good feedback because operations appear to have completed quickly and yet may not have actually taken place.

• are all operations to be available offline?

• how do we transition existing users to the new model without causing pain?

• The offline model will make "Unsent Items" more important -- should we promote it to a high-visibility UI element (like Mail.app's Outbox)?
  • Until we have a failure in sending the Outbox doesn't offer much value, however once a failure occurs it needs to be high-visibility (clarkbw)
  • Do you suggest a virtual Outbox that will show up only there are pending messages? (emre)

• What UI do we present to allow users to override the auto-download of all message bodies? Is it per folder? Per account? Global?
  • This problem needs to be broken out into what I believe are 2 separate concerns. (clarkbw)
    • Concern 1: Change. Some people will be concerned about this change no matter what and will want an "off switch" or will complain loudly.
    • Concern 2: Space. Other are probably concerned with space issues of auto-downloading, this requires some thought and is most likely the real underlying concern of #1.

• Do we give the user feedback when the delete actually happens? (davida: IMO no, only when delete fails)
  • We need to give feedback for every operation but on different levels (clarkbw)
    • Notice feedback for operations that are completed
    • Error feedback (and rollback) for operations that fail

• Do we make the Trash an offline folder like all other folders? (davida: Why not?)
  • We need to make Trash fast, responsive, and work offline (clarkbw)

• Are there other imap operations that we would want to do "offline", like renaming a folder, or should we stick with the most often used commands, like reading a message, move, delete, SMTP send, etc?
  • Why not? (clarkbw)
  • Draft Messages

Next step is to look at this from some experiences we want to happen, sending mail, receiving mail, deleting mail.

Development Strategy

1. Consider implementing what follows using a "proxy" to mediate between the UI and the IMAP protocol layer (to be reusable w/ other protocols)
2. Switch pref defaults
   • switch the default for imap folders so that they're configured for offline use out of the box. (code URL?)
   • switch mail.server.default.autosync_offline_stores to true, by default
3. Do UI operations in offline mode always
   • change imap move/deletes though the UI so that they're done offline, and the offline operation is played back after the next message is displayed, or immediately if there's no next message to display.
4. Come up with a message header and message body download strategy
   • We probably want to download recent information before older information. The most succinct form of the command to fetch multiple messages (bodies or headers) fetches them in UID, roughly chronological order, using ranges, e.g., 1-20. But you certainly could fetch 20,19,18,17 etc. Are you talking about the new profile case, or the every day case? For the new profile case, a simple thing to do would be to fetch the bodies of the unread messages first, then the bodies of the read messages. Or we could fetch ranges of messages, but start at the end, not the top, e.g., 80-100, 60-79, etc.
   • We should be careful about downloading automatically large message bodies, as they may block IMAP calls.
   • This would allow us to consider applying the same sort of logic to other protocols (NNTP?)
5. Come up with an offline operation playback strategy
   • May be worth implementing a few and seeing which ones work out best in practice.
6. Address consequences of preference changes
   • address growth of offline store folder - come up with a compacting strategy that is user-efficient. (once a week? once a month? Once a file reaches some % growth since the last compacting?)
   • Make sure that offline operations that fail give the user appropriate context as to the failure (offer to open up original message when sends fail, etc.)
7. NB: At this point, the system should be releasable
8. Consider further enhancements
   • Use the new platform code to detect whether there is a net connection or not, and 1) display that status to the user somehow, and 2) use the status to determine when to replay offline operations.
   • Use the Idle service to download message bodies when the user is idle, or to compact offline stores
   • Breakup the download of message bodies into multiple passes, which would allow the user to sneak in and start reading messages before we've synced the whole folder.
   • Playback offline operations when the user is idle, though I think we'd run into less issues if we tried to do the offline playback closer to the UI event.

Emre: Proposal for Implementation

I like to break down these requirements into 3 distinct implementation tasks:

• Offline operation playback feature implementation (needs a better name)
• Preemptive/Automatic message download feature implementation
• Background message send feature implementation

1. Offline operation playback feature Delete operation covers two different operations at the same time: Delete and Move. These two server-side operations cover the majority of use cases that require better offline support. Since it provides good granularity for deployment and testing, it makes sense to start implementing "Proxy" mechanism focusing on Delete operation first. Possible milestones are;
   1. Implement offline support for "Delete" operation:
      • Deleting messages, considering two flavors of delete;
        • Move to Trash
        • Delete immediately
      • Deleting folders
   2. Implement UI feedback mechanism (UI elements and logic)
      • consider strategies for 3 different type of errors
        • immediate errors returned by imapService->PlaybackAllOfflineOperations method. Mostly standard NSPR errors such as out of memory etc..
        • network errors that will be returned by OnStopRunningUrl callback in exit code parameter
        • server errors that will be returned by OnStopRunningUrl callback in exit code parameter
      • Modify error notification mechanism in imap protocol layer
   3. Implement offline support for other operations (rename, flags, copy, folder creation)

[Emre May 21st, 2008] bug 435153 has been filed.

1. Preemptive/Automatic message download feature Possible milestones are:
   1. Modify current offline features
      • Switch the default for imap folders so that they're configured for offline use out of the box
      • Switch mail.server.default.autosync_offline_stores to true, by default
   2. Implement automatic compacting mechanism for mbox files
   3. Re-factor current offline mechanism to implement strategy-based, automatic message body/attachment download. Couple ideas are:
      • Prioritize by date
      • Prioritize by sender
      • Prioritize by size, or attachment count
      • Idle service usage
      • Multi pass download for messages with multiple attachments
   4. Implement UI elements and logic

[Emre May 30th, 2008] 436615 bug  436615 has been filed.

1. Background message send feature Possible milestones are:
   1. Implement a "Unsent" folder UI element for IMAP folders
   2. Modify existing code to:
      • Create a new nsMsgFolder type
      • Show pending messages inside this folder - allow limited user interaction (cancel only?)
      • Don't show "sending" dialog anymore

• Shortcomings of the current implementation
  • Sends messages one by one
  • Can't send just a single message from unsent folder
  • Doesn't allow to edit a message in unsent folder

Task List for Feature 1

[Emre May 22nd, 2008] This task list is now obsolete.

1. Implement Proxy mechanism (Estimation: ?)
   • Shall map local folder model to server folder model, vice versa.
   • Shall execute commands on background - decouple folder event from protocol command.
   • Shall notify UI if the operation is not successful.
   • (?) Shall store commands persistently in case that TB exits while there are pending operations in the Q.
   • Shall deal with unsolicited UI events and IMAP server messages.
   • Shall rollback the UI operation if the command is not successful.
   • Shall be smart enough to deal with event chaining, for example; Assume that Delete mode is "MoveToTrash", the user deletes 100 messages. TB moves the messages into the trash immediately and starts the operation on the background. Then the user moves 50th deleted message back into the INBOX and mark it as "Not Read" before the "move to trash" command is sent to the server. Proxy should be smart enough not to lose the message identifier, and change the command chain if necessary - moving a message into trash might change its identifier on the server, so the content of the pending command.
2. Refactor Imap protocol layer to handle error cases gracefully, such as no-connection, server error, command execution error. Note: Put current threading model into consideration. When the user exceeds the number of cached-connections, all pending operations waiting for this session should be executed before recycling the thread. Thread recycling causes disconnection form the server which might cause an implicit EXPUNGE command on the server. (Estimation: ?)
3. Implement UI feedback mechanism (Estimation: ?)

Total:

Risks:

Decisions to make

[bienvenu] My suggestion would be to leverage the existing offline support. When the user is using TB in offline mode, we remember all the offline ops in the .msf file, and play them back when online. This gives you persistance, playback, etc.

[Emre May 22nd, 2008] I am convinced that the mechanism we need for this feature has already been implemented in the existing offline support. I am going to leverage it with bienvenu's help.

1) Should we warn the user about pending offline operations before exit?

• if yes;
  • should we give the user an option to store them locally for the next time?
  • or should we do it automatically
• if no;
  • What's the right action to take?

Even this is a rare case, it is inevitable: imagine a scenario where the user deletes 200 messages and tries to exit immediately.

[Emre]We should warn the user about pending operations and give the option to either execute them before exit TB, or save them for the next time.

[Emre May 22nd, 2008] Existing offline mechanism stores all the operations persistently in the database and playback them when TB becomes online, or in case of exit, at the startup in UpdateFolder(). AFAIK it doesn't give any kind of warning for pending operations before exit, it does silently. We are going to use the same workflow.

2) Offline operation storage:

• Is offline operation persistency required?
  • If yes, for which operations? Delete? Rename? Flag change? Tagging?
• How long should we store it?
• How to deal with aged pending offline operations?
• What kind of store we should use? Mork, Sqlite, text file
• Is security a concern - encryption?

[Emre May 22nd, 2008] No change will be done to the existing offline support in this regard.

3) UI feedback for:

• Operations that are completed
• Error feedback (and rollback) for operations that fail
• Feedback for partially or not downloaded messages (?)

[Emre May 22nd, 2008] No change will be done to the existing behavior in this regard. Error messages will be close to the user actions to prevent confusion. If the operation is failed, a message dialog will be shown (existing behavior) and the operation will be saved to be play-backed until it is successful.

4) When do we playback offline operations: [Emre] We can deal with this problem by making a design that is flexible enough to implement different strategies. Don't believe it would cost us much in term of time.

[Emre May 22nd, 2008] Two possible strategies are;

• Playback after the next messages has been fetched. Rationale: There is only one session to the server, and commands are sent one by one. Fetching the next message body before playing-back the pending operation decreases wait time for the user.
• Playback using an one-shot preset timer. Rationale: This is very similar to the first one, but it is not bound to any condition. Existing workflow is to moving/copying/deleting messages on the server then fetching the next message in the list to show. Firing a timer, lets say with a 500ms delay, to playback pending operation gives enough time to UI to fetch the next message (or to execute whatever next step is in the workflow) before the pending op is sent to the server - as I understand it, since all UI events are queued and executed by UI thread, the order of the events is more important than the delay time.

Note that in both strategies, error cases should be handled properly. For example, if the playback request is failed for some reason, TB should give an error message only the first time (clarkbw?), and should suppress the error messages for the consecutive tries.

Design Proposal

See current design [TBD]

Task List for Feature 2

Existing Behavior

Decision table for IMAP message operations as of 6th June 2008:

Copying/Moving an IMAP message

1

2

3

4

5

6

7

8

9

10

11

12

Source folder is in online mode

Y

Y

Y

Y

N

N

N

N

N

N

N

N

Destination folder is in online mode

Y

Y

N

N

Y

Y

Y

Y

N

N

N

N

TB is in online state

Y

N

Y

N

Y

Y

N

N

Y

Y

N

N

Message is available locally

-

-

-

-

Y

N

Y

N

Y

N

Y

N

Actions

Remove the header from the source folder's database

X2

X2

X2

X2

X2

X2

X2

X2

X2

X2

X2

X2

Add the new header into the destination folder's database

X1

X

X1

X

X1

X1

X

X

X1

X1

X

X

Copy the message into the destination folder's mbox

X

X

X

• DELETE and MOVE are the same operations in case that DELETE means "move to Trash"
• ¹ If and only if the destination folder is already selected (having connection to the server)
• ² Do not remove if the operation is COPY

Selecting an IMAP message	1	2	3	4
Message folder is in online mode	Y	Y	N	N
TB is in online state	Y	N	Y	N
Actions
Fetch message from the server	X	-	X	-
Get local copy if available		X		X
Store locally in the folder's mbox			X

Tagging an IMAP message	1	2	3	4
Message folder is in online mode	Y	Y	N	N
TB is in online state	Y	N	Y	N
Actions
Store the flag on	server	local db	server	local db

Selecting an IMAP folder	1	2	3	4
Message folder is in online mode	Y	Y	N	N
TB is in online state	Y	N	Y	N
Actions
Fetch message headers from the server	X		X
Remove headers from the database that do not exist on the server	X		X
Add new headers to the local database	X		X
Fetch headers from the local database		X		X
Playback offline operations	X1		X1

¹ If and only if the selected folder is the source folder of the pending offline operation

Requirements

1. TB shall give two options to the user about how to download messages when a new account is created. Options are:
   1. TB downloads all message headers and message bodies. The user has to wait until the downloads are completed.
   2. TB downloads all message headers and starts downloading message bodies on background according to a predefined strategy. if the user selects a message on the list, TB should interrupt the current download without loosing the parts already downloaded, and should start downloading the selected message.
2. When new messages arrive, TB should download message headers immediately and should start downloading the bodies if the download queue is empty. If the download queue is busy, the decision can be made based on the active strategy (see possible strategies above)
3. Highest priority should be given always to the selected message.
4. All operations conducted on IMAP folders should be completed immediately, except copying/moving IMAP messages/folders to another server or account type.
5. All messages should be stored locally for offline and fast access.
6. Message header synchronization with the IMAP server shall be done silently, on background.
7. A new error notification mechanism is needed to accommodate new UI elements.

Implementation Plan

• Requirement #1.1 can be implemented by simply enabling AutoSyncOfflineStores option and putting the selected folder into offline mode.
• Requirement #1.2 can be implemented with the help of the following new components:
  • Operation Queue; Its main duty is to serialize access to imap protocol queue, and prioritize requested online operations. The components mentioned below are the consumer of this one.
  • Message download manager; Main duties are to coordinate partial downloads, to make strategy-based decisions to prioritize messages. In the context of this component we need to implement PARTIAL FETCH commands in imap protocol level. The existing code fetches the mime structure of the message, and looks at the types of all the parts. If a message has parts TB doesn't know how to render inline (e.g., a .zip file, or a .doc or a .pdf), it fetches the body, and the parts TB knows how to display inline separately, and doesn't fetch the parts that TB can't render inline. We can definitely leverage this feature.
  • Operation playback manager; To playback pending operations on the server and handling errors gracefully. In other word activating offline operation state machine when required. This component is partially implemented as part of Offline operation playback feature.
• Requirement #1.3 is a matter of adding this option into mailnews.js
• Requirement #2 same as #1.2
• Requirement #4 can be implemented by storing all operations locally, and playing them back using Operation playback manager. To store the operations locally TB can be put to the following states (see Existing Behavior section):
  • • Copying/Moving an IMAP message: one of the states 2, 8, 11, 12
    • Selecting an IMAP message: one of the states 2, 3, 4
    • Tagging an IMAP message: 2 or 4
    • Selecting an IMAP folder: 2 or 4 since header synchronization will be done on background - see following item
  • Common divider of these states is 2, in other word, it is possible to cover every scenario for this feature by handling every IMAP operation as TB is in offline state.
• Requirement #5 will be handled when #1 and #2 are implemented. Only additional requirement for this feature is to running COMPACTING operation automatically on mbox files to keep them slim.
• Requirement #6 requires creativity to keep the local database in sync with the server (including IDLE reponses). Currently TB creates fake keys for the headers generated during an offline operation, and it replaces them (the headers) during the folder update. Possible solutions are:
  • Solving the problem at UI level as David Bienvenu once suggested "One general approach would be to have the view ignore the removal of fake keys, and the addition of new keys, and instead tell it to replace one key with an other key. But that's a pretty half-baked thought at this point. An other approach would be tell the view about a set of message-ids that this is going to happen to - then, when it gets told about a delete of message with that message-id, it would ignore it, and when it gets told about the addition of a message with that same message id, it would go find the old message with that message id, and tweak that view entry to have the new message key."
  • Adding a new column to the local database to store the server keys when available. All operations will use local keys, server keys will be use only to sync with the server and to map header to its server counterpart.
  • Using Message-Id header + other headers to uniquely identify messages in order to make "remove/add to the local database" decision.
  • Implementing a middle layer to map local keys to server keys. In other word, a component to map server data model to the local one.
• Requirement #7 can be provided by implementing new UI elements.
• One non-functional requirement is error handling. Error mechanism should be changed. Currently all imap errors are handled in imapserver. This code should be re-factored in order to make it work with new UI elements.

Resources

• Dale is working on IMAP partial fetch feature.
• Emre is working on queue mechanism.
• David works on RFC 4551. He will be consulted for design, implementation and integration issues along the way.
• No assignment for UI-side implementation yet.

Estimation

Feature-complete on June 22th. 2 weeks testing. It will be ready to ship with 3.0a2.

Risk

[TBD]

• Changes in event mechanism: Move, Delete, Copy event handling.
• Unforeseen integration problems.
• Problems in the database message key synchronization.
• Error handling problems.
• UI might not be ready by the deadline because of the resourcing constraints.

Decisions to make

• How we keep the local data model and server data model in sync?

Task List for Feature 3

[TBD]

Decisions to make

[TBD]