MozillaWiki

MailNews:Better Faster IMAP Plan: Difference between revisions

MailNews Discussion

MailNews:Better Faster IMAP Plan (view source)

Revision as of 15:09, 8 December 2008

6,123 bytes added ,  8 December 2008
m
Corrected mime_parts_on_demand truth table logic
m (Corrected mime_parts_on_demand truth table logic)
 
(27 intermediate revisions by 6 users not shown)
Line 14: Line 14:
** Until we have a failure in sending the Outbox doesn't offer much value, however once a failure occurs it needs to be high-visibility (clarkbw)  
** Until we have a failure in sending the Outbox doesn't offer much value, however once a failure occurs it needs to be high-visibility (clarkbw)  
** Do you suggest a virtual Outbox that will show up only there are pending messages? (emre)
** Do you suggest a virtual Outbox that will show up only there are pending messages? (emre)
*** Exactly, see [[Thunderbird:Folder Pane#Outbox]]


* What UI do we present to allow users to override the auto-download of all message bodies?  Is it per folder?  Per account?  Global?
* What UI do we present to allow users to override the auto-download of all message bodies?  Is it per folder?  Per account?  Global?
Line 19: Line 20:
*** Concern 1: Change. Some people will be concerned about this change no matter what and will want an "off switch" or will complain loudly.
*** Concern 1: Change. Some people will be concerned about this change no matter what and will want an "off switch" or will complain loudly.
*** Concern 2: Space.  Other are probably concerned with space issues of auto-downloading, this requires some thought and is most likely the real underlying concern of #1.
*** Concern 2: Space.  Other are probably concerned with space issues of auto-downloading, this requires some thought and is most likely the real underlying concern of #1.
*** Concern 3: Policy.  Some people may be under a policy that does not allow them to keep a local store of messages. (jaym)


* Do we give the user feedback when the delete actually happens? (davida: IMO no, only when delete fails)
* Do we give the user feedback when the delete actually happens? (davida: IMO no, only when delete fails)
Line 32: Line 34:
** Draft Messages
** Draft Messages


Next step is to look at this from some experiences we want to happen, sending mail, receiving mail, deleting mail.
 
'''Account Settings'''
The goal of the account settings is to allow people to override the global offline preference on an account by account basis.
 
<pre>
+--------------------------------------------------------------------+
|  Syncing & Disk Space                                              |
+--------------------------------------------------------------------+
|  Message Synchronization                                          |
|    [x] Keep all messages for this account on this computer        |
|                                                  ( Advanced... )  |
|                                                                    |
+--------------------------------------------------------------------+
</pre>
 
* Checkbox (defaults to global offline option)
** When checked, changes all folders in the account to be sync'd
** When unchecked changes all folders in account to not be sync'd
* ( Advanced... )
** Opens folder selector for changing individual folder setup


== Development Strategy ==
== Development Strategy ==
Line 170: Line 191:


=== 2) Preemptive/Automatic Message Download ===
=== 2) Preemptive/Automatic Message Download ===
  [Emre May 30th, 2008] {{bug| 436615}} has been filed.
  [Emre May 30th, 2008] {{bug|436615}} has been filed. Marked FIXED 2008-09-17 17:01:35 PDT.


==== Existing Behavior ====
==== Existing Behavior ====
Line 211: Line 232:
| Fetch message from the server || <center>X</center> || <center>-</center> || <center>X</center> || <center>-</center>
| Fetch message from the server || <center>X</center> || <center>-</center> || <center>X</center> || <center>-</center>
|-
|-
| Get local copy if available|| &nbsp; || <center>X</center> || &nbsp; || <center>X</center>
| Get local copy if available|| &nbsp; || <center>X</center> || <center>X</center> || <center>X</center>
|-
|-
| Store locally in the folder's <i>mbox</i> || &nbsp; || &nbsp; || <center>X</center> || &nbsp;
| Store locally in the folder's <i>mbox</i> || &nbsp; || &nbsp; || <center>X<sup>1</sup></center> || &nbsp;
|-
|-
|}
|}
*<sup>1</sup> Depending on preferences message '''''is not''''' stored locally if non-inline MIME parts (e.g. attachments) size > mime_parts_on_demand_threshold (default ~30KB) '''OR''' total message size > limit_offline_message_size (default off). See Advanced and Offline & Diskspace preferences.


   
   
Line 251: Line 273:
|}
|}
<sup>1</sup> If and only if the selected folder is the source folder of the pending offline operation
<sup>1</sup> If and only if the selected folder is the source folder of the pending offline operation
===== How it works =====
Source and Destination folders are offline, autosync_offline_stores is set to true by default, pseudo offline is enabled:
* When a folder is selected the first time, TB fetches all message keys (a.k.a Messages unique identifier values) belong to this folder from the server (using <tag> UID fetch 1:* (FLAGS) imap command). It keeps the mutual keys intact, adds new keys into the local database, and removes the non-existing ones. Once this operation is completed, it fetches the headers and message bodies belong to the new keys (I think all headers first, bodies later, not sure). Finally, it stores the next UID value to determine whether any messages have been delivered to the mailbox next time it is selected.
* If another folder is selected before all message bodies are downloaded, it stops downloading, switches to the other folder and starts the same process on this newly selected folder. Once this is completed, it resumes the download of the message bodies belong to the previously selected folder. A progress bar appears at the bottom right side of the window, but it doesn't show any progress.
* Unless it is in offline state, TB always fetches message bodies from the server even when they are available locally.
* When messages are moved from a folder to another, TB moves them immediately (Pseudo-Offline) and playback the operation later. Since TB creates fake msg keys during offline move, when the destination folder is selected, all its content are fetched again because of the reason explained above. '''mbox''' file for this folder increases as well.
* Delete is similar to Move operation if delete mode is move-to-trash.


==== Requirements ====
==== Requirements ====
Line 263: Line 298:
# A new error notification mechanism is needed to accommodate new UI elements.
# A new error notification mechanism is needed to accommodate new UI elements.


==== Implementation Plan ====
===== Message Download Strategy =====
NOT COMPLETED


{|border=1
| <b>&nbsp;</b> || <center>1</center> || <center>2</center> || <center>3</center> || <center>4</center>
|-
| Message is selected by the user || <center>Y</center> || <center>Y</center> || <center>N</center> || <center>N</center>
|-
| Sender is in the address book || <center>Y</center> || <center>N</center> || <center>Y</center> || <center>N</center>
|-
| Message date || <center>Today</center> || <center>Older</center> || <center>Y</center> || <center>N</center>
|-
| Message has attachments || <center>-</center> || <center>-</center> || <center>-</center> || <center>-</center>
|-
| Message size is .. || <center>-</center> || <center>-</center> || <center>-</center> || <center>-</center>
|-
| MIME disposition tag .. || <center>-</center> || <center>-</center> || <center>-</center> || <center>-</center>
|-
| <b><i>Actions</i></b>|| &nbsp; || &nbsp; || &nbsp; || &nbsp;
|-
| Stop current operation || &nbsp; || &nbsp; || &nbsp; || &nbsp;
|-
| Put the message into the 1st spot || &nbsp; || &nbsp; || &nbsp;|| &nbsp;
|-
| Increase the priority of the message || &nbsp; || &nbsp; || &nbsp; || &nbsp;
|-
| Decrease the priority of the message || &nbsp; || &nbsp; || &nbsp; || &nbsp;
|-
|}
==== Implementation Plan #1 (Low hanging fruit approach)====
* Requirement #1.1 can be implemented by simply enabling AutoSyncOfflineStores option and putting the selected folder into offline mode.
* Requirement #1.1 can be implemented by simply enabling AutoSyncOfflineStores option and putting the selected folder into offline mode.
* Requirement #1.2 - skip
* Requirement #2 - No strategy based download
* Requirement #4 can be implemented by putting folders into offline mode, storing all operations locally, and playing them back using <i>Playback manager</i> (see below).
* Requirement #5: putting folders in offline mode would store the messages locally, but the current workflow should be changed to fetch messages locally, instead of fetching them from the server every time.
* Requirement #6 - skip
* Requirement #7 - skip
==== Implementation Plan #2 (Ideal)====
* Requirement #1.1 can be implemented by simply enabling AutoSyncOfflineStores option and putting the selected folder into offline mode - adding this option into mailnews.js file.
* Requirement #1.2 can be implemented with the help of the following new components (please note that these are conceptual rather than physical):
* Requirement #1.2 can be implemented with the help of the following new components (please note that these are conceptual rather than physical):
** <b>Priority Queue</b>; Its main duty is to serialize access to imap protocol queue, and prioritize download and playback requests.
** <b>Priority Queue</b>; Its main duty is to serialize access to imap protocol queue, and prioritize download and playback requests.
** <b>Message download manager</b>; Main duties are to coordinate partial downloads, and to make strategy-based decisions to prioritize messages. In the context of this component we need to implement PARTIAL FETCH commands in imap protocol level. The existing code fetches the mime structure of the message, and looks at the types of all the parts. If a message has parts TB doesn't know how to render inline (e.g., a .zip file, or a .doc or a .pdf), it fetches the body, and the parts TB knows how to display inline separately, and doesn't fetch the parts that TB can't render inline. We can definitely leverage this feature.
** <b>Message download manager</b>; Main duties are to coordinate partial downloads, and to make strategy-based decisions to prioritize messages. In the context of this component we need to implement PARTIAL FETCH commands in imap protocol level. The existing code fetches the mime structure of the message, and looks at the types of all the parts. If a message has parts TB doesn't know how to render inline (e.g., a .zip file, or a .doc or a .pdf), it fetches the body, and the parts TB knows how to display inline separately, and doesn't fetch the parts that TB can't render inline. We can definitely leverage this feature.
** <b>Playback manager</b>; To playback pending offline operations on the server and handling errors gracefully. In other word activating offline operation state machine when required. This component is partially implemented as part of <i>Offline operation playback</i> feature.  
** <b>Playback manager</b>; To playback pending offline operations on the server and handling errors gracefully. In other word activating offline operation state machine when required. This component is partially implemented as part of <i>Offline operation playback</i> feature.  
* Requirement #1.3 is a matter of adding this option into mailnews.js
* Requirement #2 same as #1.2
* Requirement #2 same as #1.2
* Requirement #4 can be implemented by storing all operations locally, and playing them back using <i>Playback manager</i>. To store the operations locally TB can be put to the following states (''see Existing Behavior section''):
* Requirement #4 can be implemented by storing all operations locally, and playing them back using <i>Playback manager</i>. To store the operations locally TB can be put to the following states (''see Existing Behavior section''):
Line 277: Line 350:
*** <i>Tagging an IMAP message</i>: 2 or 4
*** <i>Tagging an IMAP message</i>: 2 or 4
*** <i>Selecting an IMAP folder</i>: 2 or 4 since header synchronization will be done on background - see following item
*** <i>Selecting an IMAP folder</i>: 2 or 4 since header synchronization will be done on background - see following item
** Common divider of these states is 2, in other word, it is possible to cover every scenario for this feature by handling every IMAP operation as TB is in offline state.
** Common denominator of these states is 2, in other word, it is possible to cover every scenario for this feature by handling every IMAP operation as TB is in offline state.
* Requirement #5 will be handled when #1 and #2 are implemented. Only additional requirement for this feature is to running COMPACTING operation automatically on mbox files to keep them slim.
* Requirement #5 will be handled when #1 and #2 are implemented. Only additional requirement for this feature is to running COMPACTING operation automatically on mbox files to keep them slim.
* Requirement #6 requires creativity to keep the local database in sync with the server (including IDLE reponses). Currently TB creates fake keys for the headers generated during an offline operation, and it replaces them (the headers) during the folder update. Possible solutions are:
* Requirement #6 requires creativity to keep the local database in sync with the server. Currently TB creates fake keys for the headers generated during an offline operation, and it replaces them (the headers) during the folder update. Possible solutions are:
** Solving the problem at UI level as David Bienvenu once suggested "One general approach would be to have the view ignore the removal of fake keys, and the addition of new keys, and instead tell it to replace one key with an other key. But that's a pretty half-baked thought at this point. An other approach would be tell the view about a set of message-ids that this is going to happen to - then, when it gets told about a delete of message with that message-id, it would ignore it, and when it gets told about the addition of a message with that same message id, it would go find the old message with that message id, and tweak that view entry to have the new message key."
*# Solving the problem at UI level as David Bienvenu once suggested "One general approach would be to have the view ignore the removal of fake keys, and the addition of new keys, and instead tell it to replace one key with an other key. But that's a pretty half-baked thought at this point. An other approach would be tell the view about a set of message-ids that this is going to happen to - then, when it gets told about a delete of message with that message-id, it would ignore it, and when it gets told about the addition of a message with that same message id, it would go find the old message with that message id, and tweak that view entry to have the new message key."
** Adding a new column to the local database to store the server keys when available. All operations will use local keys, server keys will be use only to sync with the server and to map header to its server counterpart.
*# Adding a new column to the local database to store the server keys when available. All operations will use local keys, server keys will be use only to sync with the server and to map header to its server counterpart.
** Using Message-Id header + other headers to uniquely identify messages in order to make "remove/add to the local database" decision.
*# Using Message-Id header + other headers to uniquely identify messages in order to make "remove/add to the local database" decision.
** Implementing a middle layer to map local keys to server keys. In other word, a component to map server data model to the local one.
*# Implementing a middle layer to map local keys to server keys. In other word, a component to map server data model to the local one.
** Note: UNDOing operations should be handled as well.
** Note: Operation UNDO, and IDLE responses should be handled as well.
* Requirement #7 can be provided by implementing new UI elements.   
* Requirement #7 can be provided by implementing new UI elements.   
* One non-functional requirement is error handling. Error mechanism should be changed. Currently all imap errors are handled in imapserver. This code should be re-factored in order to make it work with new UI elements.
* One non-functional requirement is error handling. Error mechanism should be changed. Currently all imap errors are handled in imapserver. This code should be re-factored in order to make it work with new UI elements.


==== Tasks ====
==== Tasks for #1====
* Playback Manager - ''partially implemented'' (?, estimate: ?)
** Activate playback state machine
** Handle errors
** Group playback request
* Change existing workflow to fetch messages from the local store
* Implement automatic compacting
 
==== Tasks for #2====
* Priority Queue implementation (?, estimate: ?)
* Priority Queue implementation (?, estimate: ?)
* Message Download Manager implementation  
* Message Download Manager implementation  
Line 300: Line 381:
** Handle errors
** Handle errors
** Group playback request
** Group playback request
* Local model to server model mapping component implementation (emre, estimate: 4 days)
* Local model to server model mapping component implementation (emre, estimate: ? days)
* Error handling mechanism implementation (?)
* Error handling mechanism implementation (?)
* UI element implementation (?)
* UI element implementation (?)
Line 306: Line 387:


==== Estimation ====
==== Estimation ====
Feature-complete on June 22th. 2 weeks testing. It will be ready to ship with 3.0a2.
''Feature-complete by June 22th(??). 2 weeks testing. It will be ready to ship with 3.0a2.(??)''.
 
I prefer to get Bienvenu's and Dale's opinions before coming with a realistic estimate.


==== Risk ====
==== Risks ====
[TBD]
[TBD]
* Changes in event mechanism: Move, Delete, Copy event handling.
* Changes in event mechanism: Move, Delete, Copy event handling.
Line 317: Line 400:


==== Decisions to make====
==== Decisions to make====
* How we keep the local data model and server data model in sync?
* Which implementation to go?
* If we go with #2, how we keep the local data model and server data model in sync?
** [emre] I prefer 4th approach. I can't say I totally understand the overall system design yet but it seems doable. Here is the rationale:
*** It doesn't require any change of the database schema
*** It hooks into the following points:
***# the point where TB retrieve the headers from the server, just before writing them into the database
***# the point where TB generates fake keys for offline operations
***# the point where UNDOing happens
***# just before the system exits, to write the real keys into the database
***# it's standalone and can be tested separately
***# (do I miss anything else?)
*** Requires no change of database implementation nor folder interaction


=== 3) Background Message Send ===
=== 3) Background Message Send ===
Nickel chrome
113

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/95890...119797"