Storage: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(add disk seek performance optimization as a proposed development priority)
(Update the page per discussion with storage advisory group)
Line 1: Line 1:
Storage is a SQLite database API.  API documentation is available in the [http://developer.mozilla.org/en/docs/Storage Mozilla Developer Center].
Storage can mean many things in Gecko and Firefox.


== Proposed Future Development Priorities ==
Possibilities include:
* mozStorage, the API that lives under [https://searchfox.org/mozilla-central/source/storage storage/] in the source tree.  This is a C++ wrapper around SQLite that is also exposed to JS via both XPCOM/XPConnect ([https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Storage API docs]) and via a friendlier JS wrapper, [https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/Sqlite.jsm SQLite.jsm].  Bug 1482608 also
* The profile "storage/" directory that holds per-origin storage managed by the QuotaManager subsystem and its clients: IndexedDB, the Cache API, and the asm.js cache (which will be removed).


=== Full Text Indexing ===
== Storage Alternatives ==
* separation of indexing from storage
** Firefox could use this to index visited web pages without having to store the contents of the pages themselves.
** Thunderbird could use this to index email messages without having to store their contents in SQLite (ultimately it might make sense to store contents as well, but that's a larger, longer-term change).
** Komodo could use this to index source code stored in files on disk.
* built-in tokenization of Unicode text (i.e. without having to either compile in a large separate library or write custom functions);
* tokenization of marked up text (HTML, PDF, etc.).


=== Performance ===
There are many options for storing data in Gecko.  The Browser Architecture Group has [https://github.com/mozilla/firefox-data-store-docs cataloged] many of the existing uses.


Dr. Hipp posted to the SQLite users list about a disk seek performance optimization that could be a boon to Places performance:
If you are looking for a way to store data in Firefox, here are the usual options:
 
* JSON flat files. This usually is not a viable option from C++ because we don't have convenient ways to manipulate JS objects from C++.
  Creating an index on A,B is equivalent to sorting on A,B.
* [https://github.com/mozilla/rkv rkv], a "simple, humane, typed Rust interface to LMDB".  LMDB is a memory-mapped flat file database.
   
* SQLite via the mozStorage API or bindings.  You should *NOT* directly access the SQLite API.  If you are using rust code and would like a better binding API than mozStorage, that makes sense, but please reach out to the Storage Advisory Group so that we can reach consensus.  We do not want to have N separate bindings to SQLite, especially since mozStorage is responsible for global configuration settings that could fight with other SQLite bindings.
The sorting algorithm currently used by SQLite requires
O(NlogN) comparisons, which is optimial. But it also requires
O(N) disk seeks, which is very suboptimal.  You don't notice
all these seeks if your database fits in cache.  But when you
get into databases of about 3GB, the seeking really slows you
down.
   
A project on our to-do list is to implement a new sorter
that uses O(1) seeks.  We know how to do this.  It is just
finding time to do the implementation.

Revision as of 15:55, 13 September 2018

Storage can mean many things in Gecko and Firefox.

Possibilities include:

  • mozStorage, the API that lives under storage/ in the source tree. This is a C++ wrapper around SQLite that is also exposed to JS via both XPCOM/XPConnect (API docs) and via a friendlier JS wrapper, SQLite.jsm. Bug 1482608 also
  • The profile "storage/" directory that holds per-origin storage managed by the QuotaManager subsystem and its clients: IndexedDB, the Cache API, and the asm.js cache (which will be removed).

Storage Alternatives

There are many options for storing data in Gecko. The Browser Architecture Group has cataloged many of the existing uses.

If you are looking for a way to store data in Firefox, here are the usual options:

  • JSON flat files. This usually is not a viable option from C++ because we don't have convenient ways to manipulate JS objects from C++.
  • rkv, a "simple, humane, typed Rust interface to LMDB". LMDB is a memory-mapped flat file database.
  • SQLite via the mozStorage API or bindings. You should *NOT* directly access the SQLite API. If you are using rust code and would like a better binding API than mozStorage, that makes sense, but please reach out to the Storage Advisory Group so that we can reach consensus. We do not want to have N separate bindings to SQLite, especially since mozStorage is responsible for global configuration settings that could fight with other SQLite bindings.
Retrieved from "https://wiki.allizom.org/index.php?title=Storage&oldid=1200997"