Confirmed users
240
edits
User:Rkent/Folder Data Persistence: Difference between revisions
User:Rkent/Folder Data Persistence (view source)
Revision as of 21:16, 2 November 2014
, 2 November 2014Conclusions for maildir folderSize
m (Folder Cache) |
m (Conclusions for maildir folderSize) |
||
| (9 intermediate revisions by the same user not shown) | |||
| Line 16: | Line 16: | ||
C:\\Users\\Kent\\AppData\\Roaming\\Thunderbird\\Profiles\\3gtxygma.default\\ImapMail\\mail.caspia.com\\Sent.msf | C:\\Users\\Kent\\AppData\\Roaming\\Thunderbird\\Profiles\\3gtxygma.default\\ImapMail\\mail.caspia.com\\Sent.msf | ||
C:\\Users\\Kent\\AppData\\Roaming\\Thunderbird\\Profiles\\3gtxygma.default\\ImapMail\\mail.caspia.com | C:\\Users\\Kent\\AppData\\Roaming\\Thunderbird\\Profiles\\3gtxygma.default\\ImapMail\\mail.caspia.com | ||
panacea.dat is owned by nsMsgAccountManager, which has methods to acquire the folder cache object and write to it. | panacea.dat is owned by nsMsgAccountManager, which has methods to acquire the folder cache object and write to it. | ||
==== Updating the Folder Cache ==== | |||
Here is an example of how the Folder Cache gets updated (along with dbFolderInfo) in a sample operation. The sample used is copying an unread message from an IMAP folder to a local folder, which will require updating the unread count and the total count for the local folder. | |||
Stack to write to local folder cache element (using esr31): | |||
<pre> | |||
> xul.dll!nsMsgLocalMailFolder::WriteToFolderCacheElem(nsIMsgFolderCacheElement * element) Line 1088 | |||
xul.dll!nsMsgDBFolder::WriteToFolderCache(nsIMsgFolderCache * folderCache, bool deep) Line 1386 | |||
xul.dll!nsMsgDBFolder::FlushToFolderCache() Line 1362 | |||
xul.dll!nsMsgDBFolder::UpdateSummaryTotals(bool force) Line 4178 | |||
xul.dll!nsMsgDBFolder::EnableNotifications(int notificationType, bool enable, bool dbBatching) Line 5085 | |||
xul.dll!nsMsgLocalMailFolder::EndCopy(bool aCopySucceeded) Line 2447 | |||
xul.dll!nsCopyMessageStreamListener::EndCopy(nsISupports * url, tag_nsresult aStatus) Line 114 | |||
xul.dll!nsCopyMessageStreamListener::OnStopRequest(nsIRequest * request, nsISupports * ctxt, tag_nsresult aStatus) Line 144 | |||
xul.dll!nsImapCacheStreamListener::OnStopRequest(nsIRequest * request, nsISupports * aCtxt, tag_nsresult aStatus) Line 8629 | |||
</pre> | |||
The method UpdateSummaryTotals is key to keeping the folder cache current, but the way information is managed there is quite convoluted. It does the following things: | |||
* Call ReadDBFolderInfo(force), which makes sure that the folder object member variables have been initialized from the cache. If force == true, or the initialization from the cache fails, then the variables are initialized from dbFolderInfo instead. So this method is badly misnamed, it should be something like "InitializeFolderMetadata". Because force==true will cause the folder db to be opened, it is important that force==true is only used in cases where the db is already open, or we expect it to be opened. If force==false, except at initialization ReadDBFolderInfo is a noop. | |||
* Sends OnItemIntPropertyChanged notifications to nsIFolderListener objects for the unread count and total count for the folder | |||
* Writes updated information to the folder cache | |||
If force==true, then things are really confusing and convoluted. In ReadDBFolderInfo, certain folder metadata (mNumTotalMessages, mNumUnreadMessages, mExpungedBytes, mName, mCharset, mCharsetOverride, nsMsgFolderFlags::GotNew) are read from dbFolderInfo (overwriting any local values in the folder object). Cache element metadata mNumPendingUnreadMessages, mNumPendingTotalMessages, mFolderSize, mFlags is not. The handling of pending counts is particularly confusing. They seem to be mostly managed through the method ChangeNumPending... which updates mNumPendingUnreadMessages and then updates dbFolderInfo but not the cache, as well as notifies. (This seems like a performance issue. You should not have to open a folder database to record that there are messages pending that have not been downloaded). | |||
How to understand all of this? The philosophy seems to be the following: | |||
1) Any time that folder metadata is changed, that change needs to be written immediately to dbFolderInfo. | |||
2) dbFolderInfo is maintained by the db, so adding a message header will implicitly update dbFolderInfo. | |||
3) db operations do message db listener notifications, but not folder-level notifications. folderCache is considered | |||
a folder-level notification, so is done by the folder. | |||
=== Updating unread count in dbfolderinfo === | |||
Stack to update unread count: | |||
<pre> | |||
xul.dll!nsDBFolderInfo::ChangeNumUnreadMessages(int delta) Line 517 | |||
xul.dll!nsMsgDatabase::AddNewHdrToDB(nsIMsgDBHdr * newHdr, bool notify) Line 3508 | |||
xul.dll!nsMsgLocalMailFolder::EndCopy(bool aCopySucceeded) Line 2378 | |||
xul.dll!nsCopyMessageStreamListener::EndCopy(nsISupports * url, tag_nsresult aStatus) Line 114 | |||
xul.dll!nsCopyMessageStreamListener::OnStopRequest(nsIRequest * request, nsISupports * ctxt, tag_nsresult aStatus) Line 144 | |||
xul.dll!nsImapCacheStreamListener::OnStopRequest(nsIRequest * request, nsISupports * aCtxt, tag_nsresult aStatus) Line 8629 | |||
</pre> | |||
So nsCopyMessageStreamListener is managing the calls. EndCopy is called first, which adds the message to the database, resulting in incrementing numUnreadMessages in dbFolderInfo | |||
=== Updating folderSize in dbFolderInfo === | |||
Stack: | |||
<pre> | |||
xul.dll!nsDBFolderInfo::SetFolderSize(unsigned __int64 size) Line 423 | |||
xul.dll!nsMsgBrkMBoxStore::SetSummaryFileValid(nsIMsgFolder * aFolder, nsIMsgDatabase * aDB, bool aValid) Line 301 | |||
xul.dll!nsMailDatabase::SetSummaryValid(bool aValid) Line 126 | |||
xul.dll!nsMsgLocalMailFolder::OnCopyCompleted(nsISupports * srcSupport, bool moveCopySucceeded) Line 1352 | |||
xul.dll!nsMsgLocalMailFolder::EndCopy(bool aCopySucceeded) Line 2455 | |||
xul.dll!nsCopyMessageStreamListener::EndCopy(nsISupports * url, tag_nsresult aStatus) Line 114 | |||
xul.dll!nsCopyMessageStreamListener::OnStopRequest(nsIRequest * request, nsISupports * ctxt, tag_nsresult aStatus) Line 144 | |||
xul.dll!nsImapCacheStreamListener::OnStopRequest(nsIRequest * request, nsISupports * aCtxt, tag_nsresult aStatus) Line 8629 | |||
</pre> | |||
== Analysis == | |||
As a general rule, changes to folder metadata is first written to dbFolderInfo without doing changes on the equivalent member variables in the msgFolder. The variables in the msgFolder are changed at the end of operations, reading from dbFolderInfo in ReadDBFolderInfo. | |||
=== ReadDBFolderInfo(force) === | |||
nsMsgDBFolder::ReadDBFolderInfo(bool force) when force==false is a no-op except for the first time a folder is created, where the folder member objects for folder metadata are initialized from the folder cache. | |||
ReadDBFolderInfo(false) is used typically in a Get...() call, where you want to make sure that the variable has been initialized from the cache before returning it. | |||
ReadDBFolderInfo(true) is used typically after folder metadata has changed in dbFolderInfo, and you want to update the folder member variables, typically also doing any required notifications. See nsMsgDBFolder::UpdateSummaryTotals | |||
=== UpdateSummaryTotals(force) === | |||
In all cases, UpdateSummaryTotals will: | |||
* Initialize member variables using ReadDBFolderInfo(force) | |||
* Notify changes in kTotalMessagesAtom and kTotalUnreadMessagesAtom | |||
* call FlushToFolderCache to update the cache. | |||
When force==true, then the objects are read from dbFolderInfo, so this is the method used to update relevant member variables by reading from dbFolderInfo, doing folder-level notifications, and flushing the changes to the folderCache. | |||
When force==false the situation is trickier. Member variables are initialized from the folder cache, with notifications, without opening the database. This is done when the folder object is first initialized (in GetSubfolders), or in SummaryChanged(). | |||
=== SummaryChanged() === | |||
This is just a synonym for UpdateSummaryTotals(false), and is only used in IMAP. Its main purpose seems to be to write folder metadata to the folder cache. This assumes that the metadata was written to dbFolderInfo at the time it was changed. One common use seems to be after ChangeNumPending... which does changes in dbFolderInfo and notifications but not in folderCache. The call to UpdateSummaryTotals(false) only flushes the changes to the folder cache. | |||
== folderSize and IMAP databases == | |||
=== Overloaded meaning and definition of folderSize === | |||
One of the things that makes folderSize so complex is that its meaning is overloaded, with at least three different uses: | |||
* On local folders with mbox, folderSize is used to report to the UI the size of the message folder on disk, as well as used as an indicator of whether the summary file is valid. Although these values are the same, the timing issues on updates for these two issues may be different. | |||
* For maildir, calculating folderSize is slow directly from the disk, but it is not used as an indicator of validity of the message summary file. | |||
* On IMAP, the summary file is still used with offline folders, but the meaning of folderSize is changed to represent the server-side storage used for messages. For this reason, folderSize is unavailable for use with offline folders to represent the validity of the summary file. | |||
How does IMAP use the summary file and mbox storage, and avoid the paths that check for summary file valid? | |||
For a local folder with mbox, toggling a message as read results in marking the summary file valid through this stack: | |||
<pre> | |||
xul.dll!nsMailDatabase::SetSummaryValid(bool aValid) Line 119 | |||
xul.dll!nsMailDatabase::EndBatch() Line 78 | |||
xul.dll!nsMsgDBFolder::EnableNotifications(int notificationType, bool enable, bool dbBatching) Line 5085 | |||
xul.dll!nsMsgDBView::ApplyCommandToIndices(int command, unsigned int * indices, int numIndices) Line 2932 | |||
xul.dll!nsMsgDBView::CycleCell(int row, nsITreeColumn * col) Line 2067 | |||
</pre> | |||
The same operation on IMAP has an overridden EndBatch which is a no-op: | |||
<pre> | |||
xul.dll!nsImapMailDatabase::EndBatch() Line 71 | |||
xul.dll!nsMsgDBFolder::EnableNotifications(int notificationType, bool enable, bool dbBatching) Line 5085 | |||
xul.dll!nsMsgDBView::ApplyCommandToIndices(int command, unsigned int * indices, int numIndices) Line 2932 | |||
xul.dll!nsMsgDBView::CycleCell(int row, nsITreeColumn * col) Line 2067 | |||
</pre> | |||
But also, SetSummaryValid for IMAP is different, without the folderSize test: | |||
<pre> | |||
NS_IMETHODIMP nsImapMailDatabase::SetSummaryValid(bool valid) | |||
{ | |||
if (m_dbFolderInfo) | |||
{ | |||
m_dbFolderInfo->SetVersion(valid ? GetCurVersion() : 0); | |||
Commit(nsMsgDBCommitType::kLargeCommit); | |||
} | |||
return NS_OK; | |||
} | |||
</pre> | |||
Where do we use summaryValid? | |||
* [http://mxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp#198 in nsMsgFolderCompactor to skip compacting folders that are being parsed] | |||
* [http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1267 on nsMsgDatabase::CheckForErrors, which is used in checking for errors when opening databases] | |||
* In MoveMail, LocalFolders, and POP3Server (all POP3/local) | |||
So for non-local folders, this is only used when opening the folder. | |||
== Conclusions for maildir folderSize == | |||
* maildir needs to send folderSize updates to the msgFolder, but IMAP needs to ignore those. That means that IMAP needs to override SetSizeOnDisk with a noop. Perhaps it needs instead SetSizeOnServer which it will use instead. | |||