MailNews:Better Faster IMAP Plan

Leverage Offline capabilities to make online more responsive

The basic idea is to use the offline capabilities of Thunderbird while online to make the UI more responsive, when operating on IMAP messages, out of the box, without the user having to tweak any settings. (Note: this applies to SMTP sending as well)

UX Decisions to make

With an assumption that all operations are to be perceived as more responsive, many of our decisions are made for us. A proxy increases speed also increase the need for good feedback because operations appear to have completed quickly and yet may not have actually taken place.

• are all operations to be available offline?

• how do we transition existing users to the new model without causing pain?

• The offline model will make "Unsent Items" more important -- should we promote it to a high-visibility UI element (like Mail.app's Outbox)?
  • Until we have a failure in sending the Outbox doesn't offer much value, however once a failure occurs it needs to be high-visibility (clarkbw)
  • Do you suggest a virtual Outbox that will show up only there are pending messages? (emre)

• What UI do we present to allow users to override the auto-download of all message bodies? Is it per folder? Per account? Global?
  • This problem needs to be broken out into what I believe are 2 separate concerns. (clarkbw)
    • Concern 1: Change. Some people will be concerned about this change no matter what and will want an "off switch" or will complain loudly.
    • Concern 2: Space. Other are probably concerned with space issues of auto-downloading, this requires some thought and is most likely the real underlying concern of #1.

• Do we give the user feedback when the delete actually happens? (davida: IMO no, only when delete fails)
  • We need to give feedback for every operation but on different levels (clarkbw)
    • Notice feedback for operations that are completed
    • Error feedback (and rollback) for operations that fail

• Do we make the Trash an offline folder like all other folders? (davida: Why not?)
  • We need to make Trash fast, responsive, and work offline (clarkbw)

• Are there other imap operations that we would want to do "offline", like renaming a folder, or should we stick with the most often used commands, like reading a message, move, delete, SMTP send, etc?
  • Why not? (clarkbw)
  • Draft Messages

Next step is to look at this from some experiences we want to happen, sending mail, receiving mail, deleting mail.

Development Strategy

1. Consider implementing what follows using a "proxy" to mediate between the UI and the IMAP protocol layer (to be reusable w/ other protocols)
2. Switch pref defaults
   • switch the default for imap folders so that they're configured for offline use out of the box. (code URL?)
   • switch mail.server.default.autosync_offline_stores to true, by default
3. Do UI operations in offline mode always
   • change imap move/deletes though the UI so that they're done offline, and the offline operation is played back after the next message is displayed, or immediately if there's no next message to display.
4. Come up with a message header and message body download strategy
   • We probably want to download recent information before older information. The most succinct form of the command to fetch multiple messages (bodies or headers) fetches them in UID, roughly chronological order, using ranges, e.g., 1-20. But you certainly could fetch 20,19,18,17 etc. Are you talking about the new profile case, or the every day case? For the new profile case, a simple thing to do would be to fetch the bodies of the unread messages first, then the bodies of the read messages. Or we could fetch ranges of messages, but start at the end, not the top, e.g., 80-100, 60-79, etc.
   • We should be careful about downloading automatically large message bodies, as they may block IMAP calls.
   • This would allow us to consider applying the same sort of logic to other protocols (NNTP?)
5. Come up with an offline operation playback strategy
   • May be worth implementing a few and seeing which ones work out best in practice.
6. Address consequences of preference changes
   • address growth of offline store folder - come up with a compacting strategy that is user-efficient. (once a week? once a month? Once a file reaches some % growth since the last compacting?)
   • Make sure that offline operations that fail give the user appropriate context as to the failure (offer to open up original message when sends fail, etc.)
7. NB: At this point, the system should be releasable
8. Consider further enhancements
   • Use the new platform code to detect whether there is a net connection or not, and 1) display that status to the user somehow, and 2) use the status to determine when to replay offline operations.
   • Use the Idle service to download message bodies when the user is idle, or to compact offline stores
   • Breakup the download of message bodies into multiple passes, which would allow the user to sneak in and start reading messages before we've synced the whole folder.
   • Playback offline operations when the user is idle, though I think we'd run into less issues if we tried to do the offline playback closer to the UI event.

Emre: Proposal for Implementation

I like to break down these requirements into 3 distinct implementation tasks:

• Offline operation playback feature implementation (needs a better name)
• Preemptive/Automatic message download feature implementation
• Background message send feature implementation

1. Offline operation playback feature Delete operation covers two different operations at the same time: Delete and Move. These two server-side operations cover the majority of use cases that require better offline support. Since it provides good granularity for deployment and testing, it makes sense to start implementing "Proxy" mechanism focusing on Delete operation first. Possible milestones are;
   1. Implement offline support for "Delete" operation:
      • Deleting messages, considering two flavors of delete;
        • Move to Trash
        • Delete immediately
      • Deleting folders
   2. Implement UI feedback mechanism (UI elements and logic)
      • consider strategies for 3 different type of errors
        • immediate errors returned by imapService->PlaybackAllOfflineOperations method. Mostly standard NSPR errors such as out of memory etc..
        • network errors that will be returned by OnStopRunningUrl callback in exit code parameter
        • server errors that will be returned by OnStopRunningUrl callback in exit code parameter
      • Modify error notification mechanism in imap protocol layer
   3. Implement offline support for other operations (rename, flags, copy, folder creation)

[Emre May 21st, 2008] bug 435153 has been filed.

1. Preemptive/Automatic message download feature Possible milestones are:
   1. Modify current offline features
      • Switch the default for imap folders so that they're configured for offline use out of the box
      • Switch mail.server.default.autosync_offline_stores to true, by default
   2. Implement automatic compacting mechanism for mbox files
   3. Re-factor current offline mechanism to implement strategy-based, automatic message body/attachment download. Couple ideas are:
      • Prioritize by date
      • Prioritize by sender
      • Prioritize by size, or attachment count
      • Idle service usage
      • Multi pass download for messages with multiple attachments
   4. Implement UI elements and logic

[Emre May 30th, 2008] 436615 bug  436615 has been filed.

1. Background message send feature Possible milestones are:
   1. Implement a "Unsent" folder UI element for IMAP folders
   2. Modify existing code to:
      • Create a new nsMsgFolder type
      • Show pending messages inside this folder - allow limited user interaction (cancel only?)
      • Don't show "sending" dialog anymore

• Shortcomings of the current implementation
  • Sends messages one by one
  • Can't send just a single message from unsent folder
  • Doesn't allow to edit a message in unsent folder

Task List for Feature 1

[Emre May 22nd, 2008] This task list is now obsolete.

1. Implement Proxy mechanism (Estimation: ?)
   • Shall map local folder model to server folder model, vice versa.
   • Shall execute commands on background - decouple folder event from protocol command.
   • Shall notify UI if the operation is not successful.
   • (?) Shall store commands persistently in case that TB exits while there are pending operations in the Q.
   • Shall deal with unsolicited UI events and IMAP server messages.
   • Shall rollback the UI operation if the command is not successful.
   • Shall be smart enough to deal with event chaining, for example; Assume that Delete mode is "MoveToTrash", the user deletes 100 messages. TB moves the messages into the trash immediately and starts the operation on the background. Then the user moves 50th deleted message back into the INBOX and mark it as "Not Read" before the "move to trash" command is sent to the server. Proxy should be smart enough not to lose the message identifier, and change the command chain if necessary - moving a message into trash might change its identifier on the server, so the content of the pending command.
2. Refactor Imap protocol layer to handle error cases gracefully, such as no-connection, server error, command execution error. Note: Put current threading model into consideration. When the user exceeds the number of cached-connections, all pending operations waiting for this session should be executed before recycling the thread. Thread recycling causes disconnection form the server which might cause an implicit EXPUNGE command on the server. (Estimation: ?)
3. Implement UI feedback mechanism (Estimation: ?)

Total:

Risks:

Decisions to make

[bienvenu] My suggestion would be to leverage the existing offline support. When the user is using TB in offline mode, we remember all the offline ops in the .msf file, and play them back when online. This gives you persistance, playback, etc.

[Emre May 22nd, 2008] I am convinced that the mechanism we need for this feature has already been implemented in the existing offline support. I am going to leverage it with bienvenu's help.

1) Should we warn the user about pending offline operations before exit?

• if yes;
  • should we give the user an option to store them locally for the next time?
  • or should we do it automatically
• if no;
  • What's the right action to take?

Even this is a rare case, it is inevitable: imagine a scenario where the user deletes 200 messages and tries to exit immediately.

[Emre]We should warn the user about pending operations and give the option to either execute them before exit TB, or save them for the next time.

[Emre May 22nd, 2008] Existing offline mechanism stores all the operations persistently in the database and playback them when TB becomes online, or in case of exit, at the startup in UpdateFolder(). AFAIK it doesn't give any kind of warning for pending operations before exit, it does silently. We are going to use the same workflow.

2) Offline operation storage:

• Is offline operation persistency required?
  • If yes, for which operations? Delete? Rename? Flag change? Tagging?
• How long should we store it?
• How to deal with aged pending offline operations?
• What kind of store we should use? Mork, Sqlite, text file
• Is security a concern - encryption?

[Emre May 22nd, 2008] No change will be done to the existing offline support in this regard.

3) UI feedback for:

• Operations that are completed
• Error feedback (and rollback) for operations that fail
• Feedback for partially or not downloaded messages (?)

[Emre May 22nd, 2008] No change will be done to the existing behavior in this regard. Error messages will be close to the user actions to prevent confusion. If the operation is failed, a message dialog will be shown (existing behavior) and the operation will be saved to be play-backed until it is successful.

4) When do we playback offline operations: [Emre] We can deal with this problem by making a design that is flexible enough to implement different strategies. Don't believe it would cost us much in term of time.

[Emre May 22nd, 2008] Two possible strategies are;

• Playback after the next messages has been fetched. Rationale: There is only one session to the server, and commands are sent one by one. Fetching the next message body before playing-back the pending operation decreases wait time for the user.
• Playback using an one-shot preset timer. Rationale: This is very similar to the first one, but it is not bound to any condition. Existing workflow is to moving/copying/deleting messages on the server then fetching the next message in the list to show. Firing a timer, lets say with a 500ms delay, to playback pending operation gives enough time to UI to fetch the next message (or to execute whatever next step is in the workflow) before the pending op is sent to the server - as I understand it, since all UI events are queued and executed by UI thread, the order of the events is more important than the delay time.

Note that in both strategies, error cases should be handled properly. For example, if the playback request is failed for some reason, TB should give an error message only the first time (clarkbw?), and should suppress the error messages for the consecutive tries.

Design Proposal

See current design [TBD]

Task List for Feature 2

Existing Behavior

Decision table for IMAP message operations. Current State (6th June 2008):

Copying/Moving an IMAP message

1

2

3

4

5

6

7

8

9

10

11

12

Source folder is in online mode

Y

Y

Y

Y

N

N

N

N

N

N

N

N

Destination folder is in online mode

Y

Y

N

N

Y

Y

Y

Y

N

N

N

N

TB is in online state

Y

N

Y

N

Y

Y

N

N

Y

Y

N

N

Message is available locally

-

-

-

-

Y

N

Y

N

Y

N

Y

N

Actions

Remove the header from the source folder's database

X2

X2

X2

X2

X2

X2

X2

X2

X2

X2

X2

X2

Add the new header into the destination folder's database

X1

X

X1

X

X1

X1

X

X

X1

X1

X

X

Copy the message into the destination folder's mbox

X

X

X

• DELETE and MOVE are the same operations in case that DELETE means "move to Trash"
• ¹ If and only if the destination folder is already selected (having connection to the server)
• ² Do not remove if the operation is COPY

Selecting an IMAP message	1	2	3	4
Message folder is in online mode	Y	Y	N	N
TB is in online state	Y	N	Y	N
Actions
Fetch message from the server	X	-	X	-
Get local copy if available		X		X
Store locally in the folder's mbox			X

Tagging an IMAP message	1	2	3	4
Message folder is in online mode	Y	Y	N	N
TB is in online state	Y	N	Y	N
Actions
Store the flag on	server	local db	server	local db

Selecting an IMAP folder	1	2	3	4
Message folder is in online mode	Y	Y	N	N
TB is in online state	Y	N	Y	N
Actions
Fetch message headers from the server	X		X
Remove headers from the database that do not exist on the server	X		X
Add new headers to the local database	X		X
Fetch headers from the local database		X		X
Playback offline operations	X1		X1

¹ If and only if the selected folder is the source folder of the pending offline operation

Implementation Plan

• Background preemptive message downloading:
  • AutoSyncOfflineStores parameter is used in nsImapMailFolder.cpp (nsImapMailFolder::HeaderFetchCompleted method) to provide automatic message download. We are going to leverage this functionality.
  • Implement a queue to provide with a strategy based fetch mechanism that also supports partial fetching
• Sync'ing with the server should be handled on background:
  • Pending offline operations should be played back regularly. This mechanism is already implemented as the part of the first feature.
  • IDLE message should be handled silently
  • Message key changes should be handled silently (without generating UI events but not changing the execution flow)
• Error mechanism should be changed. Currently all imap errors are handled in imapserver. This code should be re-factored in order to make it work with new UI elements.
• Automatic compacting of mbox files.
  • Can be done when the queue is empty and the system is idle
• UI element implementation: Waiting for design to be completed.

Resources

• Dale is working on IMAP partial fetch
• Emre is working on queueu mechanism
• David is consulted on integration, design and implementation issues. He also works on RFC 4551.
• No assignment for UI-side implementation yet.

Estimation

Feature complete on June 22th. It will be ready to ship with 3.0a2.

Risk

• Changes in event mechanism: Move, Delete, Copy event handling
• Problems in database message key synchronization
• Error handling problems
• UI might not be ready by the deadline

[TBD]

Decisions to make

[TBD]

Task List for Feature 3

[TBD]

Decisions to make

[TBD]