MozillaWiki

NSS Shared DB: Difference between revisions

Page Discussion

NSS Shared DB (view source)

Revision as of 07:20, 9 November 2007

3,700 bytes added ,  9 November 2007
partial editorial pass - some wordsmithing - more to be done
(partial editorial pass - some wordsmithing - more to be done)
Line 1: Line 1:
== Shared Database Proposal ==
== Shared Database Proposal ==


NSS has not updated it's database format since 199x. Over the years the restrictions created by the current database has become more accute. Of these restrictions, the lack of the ability for applications to share certificate and key database has been one of the more severe. As more applications use NSS, the need for each one to manage a database separately becomes unnecessary overhead.
NSS has been using an old version of the Berkeley DataBase as its database engine since Netscape Navigator 2.0 in 1994.  That version of the database engine is commonly described in NSS documents as "DBM".  That engine has a
number of limitations. One of the most severe limitations concerns the  
number of processes that may share a database file.  While any process has
a DBM file open for writing, NO other process may access it in any way. 
Multiple processes may share a DBM database ONLY if they ALL access it
READ-ONLY. Processes cannot share a DBM database file if ANY of them wants
to update it.


In 2001 we built some tools to work around those restrictions so certain applications could share the database if they supplied their own shared database implementation, and configured NSS to use that implementation. Today we have a process level, ACID, open source, and widely available database called SQLite. In addition, there is a strong desire to make NSS the system security service for Linux. I am proposing how we could leverage this database to give all of our applications Shared Database access.
This limitation has been cumbersome for applications that wish to use NSS. 
Applications that want to share databases have resorted to these strategies:


Besides the lack of shared database, other issues with the current NSS database scheme include:
*Synchronized updates, with application down time: The applications share the database read-only.  If any update is desired, all the applications are shut down, and a database update program performs the update, then all the applications are restarted in read-only mode.  Some server products, for example, have an administration program that stops the servers, updates the database that they share, and then restarts the servers.  This results in undesirable downtime and desired database changes are delayed until the next interval in which such downtime is acceptable.
# The need for a more flexible schema which can handle storing more meta information about certificates and keys, particular finer grain information about the trust of a certificate.
*Multiple copies with duplicated updates.  Each application keeps its own copy of its databases, and applications communicate their changes to each other, so that each application may apply received changes to its own DB. FireFox and Thunderbird are examples of this.  When one of those applications gets a new certificate and private key, the user may "export" that pair to a PKCS#12 file, and then import that file into the other application.  Most users never master these steps, and so have databases entirely out of sync.
# The need to match the underlying certificate and key storage with it's reflection into NSS (that is PKCS #11).
 
# FIPS requires integrity checks on critical keys and trust objects. We need to be able to store that information in the database.
These workarounds for the DBM engine's limitations are sufficiently onerous
that they prevent many applications from adopting NSS.  The desire to make
NSS more ubiquitous now motivates the elimination of these limitations. There is a strong desire to make NSS be the native OS network security service for Linux.
 
In 2001 NSS was modified to enable applications to supply their own database engines.  Applications could share a common database if they supplied their
own shared database implementation, and configured NSS to use it. 
 
Today, there exists a process level, ACID, open source, and widely available database engine with which multiple processes may simultaneous have read and write access to a shared database.  It is named SQLite.  The NSS team proposes to leverage this database to give all NSS-based applications Shared Database access.
 
Besides the inability to share databases, other issues with NSS DBM database scheme include:
# A more flexible schema which can store meta information about certificates and keys, such as finer grained information about trust for certificates.
# The need to match the underlying certificate and key storage with its reflection into NSS (that is, PKCS #11).
# To satisfy the FIPS requirements for integrity checks on keys and trust objects, NSS must be able to store integrity information in the databases.


=== Where we are today ===
=== Where we are today ===


At initialization time, NSS currently takes an argument which points to some directory the application uses to store its private configuration data. NSS uses 3 libdbm files in that directory:
At initialization time, the application gives NSS a string that it uses as the pathname of a directory to store NSS's security and configuration data. NSS typically stores 3 dbm files in that directory:
* cert8.db - stores publicly accessible objects (certs, CRLs, S/MIME records).
 
* key3.db - stores the private keys.
* cert8.db - stores publicly accessible objects (certs, CRLs, S/MIME records).
* key3.db   - stores the private keys.
* secmod.db - stores the PKCS #11 module configuration.
* secmod.db - stores the PKCS #11 module configuration.


In addition:<br />
Also in that directory:<br />
* NSS may also use a directory called cert8.dir to store very large blobs (typically large CRLs).
* If it has very large security objects (such as large CRLs), NSS will store them in files in a subdirectory named cert8.dir.
* NSS may read from from previous certificate database (cert7.db, cert5.db, etc.) and build a new cert8.db if it doesn't exist.
* If the cert8.db and/or key3.db files are missing, NSS will read data from older versions of those databases (e.g., cert7.db, cert5.db, if they exist)  
and may build new cert8.db and/or key3.db files.
 
These files are all accessed exclusively by the softoken shared library, making it the only NSS library that must be linked with libdbm.


These files are all accessed through the softoken, making libsoftokn3.so the only NSS library that needs to link with libdbm.
=== The application-supplied database feature ===


If the directory argument passed to NSS starts with the string 'multiaccess:', NSS does not use it as a directory path. Instead, NSS parses the string out as follows:
If the initialization string given to NSS starts with 'multiaccess:', NSS does not use it as a directory pathname. Instead, NSS parses the string as follows:


'''multiaccess:'''''appName''[''':'''''directory'']
'''multiaccess:'''''appName''[''':'''''directory'']


Where:<br/>
Where:<br/>
''multicaccess'' is a keyword.<br/>
''multiaccess'' is a keyword.<br/>
''appName'' is a unique string for each group of applications which share the database.<br />
''appName'' uniquely identifies a group of applications which share an
''directory'' is an optional parameter pointing to an NSS non-shared database which NSS will use to update the shared database from on loading.
application-supplied database, effectively a new database name.<br />
''directory'' is the pathname for a directory containing NSS DBM databases
whose contents will be used to update the application-supplied database during NSS initialization.
 
In the presence of a multiaccess initialization string, during initialization
NSS will try to find a shared database named librdb.so (rdb.dll on Windows) in its path and load it. This shared library is expected to implement a superset of the old dbm interface.  The main entry point is rdbopen, which will be passed the appName, database name, and open flags. The rdb shared library will pick a location or method to store the database (it may not necessarily be a file), then handle the raw db records from NSS.  The records passed to and from this library use exactly the same schema and record formats as the records in the DBM library.


NSS will find librdb.so (rdb.dll on Windows) in its path and load it. This shared library is expected to implement a superset of the standard libdbm interface. The main entry point is rdbopen, which will pass the appName, database name, and open flags. The rdb shared library will pick a location or method to store the database (it may not necessarily be a file), then handle the raw db records from NSS. The library does not do any formatting of the data.
=== The proposal ===


=== What we want to do ===
We propose to replace key3.db and cert8.db with new SQL databases called key4.db and cert9.db. These new databases will store PKCS #11 token objects, the same types of objects whose contents are currently stored in cert8.db and key3.db


We want to move key3 and cert8 into new SQL databases called key4.db and cert9.db. These databases will store the PKCS #11 objects currently stored or implied in cert8 and key3.
Optionally the new databases could be combined into a single database, cert9.db, where private and public objects are stored in separate tables. Softoken would automatically identify any cert9.db that also has an embedded key store and open that key store up, instead of opening a separate key4.db and cert9.db. However, in the first release, there will be no way to create a cert9.db containing both cert and key tables.


Optionally the databases could be combined into a single database, cert9.db, where private and public objects are stored in separate tables. Softoken will automatically identify any cert9.db which has an imbedded key store and open that key store up. By default a separate key4.db and cert9.db will be opened. In the first release there will be no way to create a combined cert/keydb.
The new databases will be called 'shareable' databases. They may or may not be shared by multiple processes, but they are all capable of being shared.


==== schema ====
==== schema ====
Line 47: Line 75:
*The Attribute values will be stored as binary blobs.
*The Attribute values will be stored as binary blobs.
**Attributes that represent CK_ULONG values will be stored as 32-bit values in network byte order.
**Attributes that represent CK_ULONG values will be stored as 32-bit values in network byte order.
**All other objects, byte order is already specified by PKCS #11.
**For all other attributes, byte order is already specified by PKCS #11.
**Private attributes will be encrypted with a PKCS #5 PBE in the same way the pkcs8 private and secret key data is encrypted today.
**Private attributes will be encrypted with a PKCS #5 PBE in the same way the pkcs8 private and secret key data is encrypted today.
**Softoken will only set those attributes appropriate for the given object. If The attribute is not appropriate it will be left blank. (Note that sqlite does not distinguish between a NULL attribute and an empty one. This will be handled by storing a special value which means 'NULL' when writing a NULL record.
**Softoken will only set those attributes appropriate for the given object. If The attribute is not appropriate it will be left blank. (Note that sqlite does not distinguish between a NULL attribute and an empty one. This will be handled by storing a special value which means 'NULL' when writing a NULL record.
** integrity will be maintained by a PBE based MAC on critical attributes.
** integrity will be maintained by a PBE based MAC on critical attributes.


Out of band data necessary for the proper operation of softoken, but not reflected back to NSS in general (currently this only applies to password entries), will be stored in separate tables in the database with their own schema.
Other data that is necessary for the proper operation of softoken, but that is not defined as part of any PKCS#11 objects, (such as data used to verify token user passwords), will be stored in separate tables in the database with their own schemas.


Database extension is accomplished in 2 ways:
Database extension will be accomplished in 2 ways:
# New attributes are added to the list known attributes and to already defined PKCS #11 objects. Older database objects can be detected because they will have 'invalid' values for these attributes (example, could add CKA_TRUST_EXTENSION_OVERRIDE to trust objects to add or override existing certificate extensions).
# New attribute types can augment the already-implemented attribute types for objects already implemented in softoken. Attributes of these new types can be added to older database objects, which will be detected because they will have 'invalid' values for these attributes.  For example, we could add a new attribute type to hold additional extensions for certificate objects.
# Add new PKCS #11 objects to hold the data (example, could add a new SSL_DATA record to store mappings to various certificates to different cipher suites and host name*)
# Define new PKCS #11 object types.  For example, we could add new objects to store mappings between various certificates pairs of cipher suites and host names.


Softoken will be able to store the following objects and attributes. In the table below, attributes marked CK_ULONG will be written to the database as a 32-bit network order unsigned integer, attributes marked 'encrypted' will be encrypted with the token's pbe key, and attributes marked 'MACed' will be MACed with the token's pbe key.
Softoken will be able to store the following objects and attributes. In the table below, attributes marked CK_ULONG will be written to the database as a 32-bit network byte order unsigned integers.  Attributes marked 'encrypted' will be encrypted with the token's pbe key, and attributes marked 'MACed' will be MACed with the token's PBE key.
===== Legal Attributes and objects =====
===== Legal Attributes and objects =====


While the key and certificate database format is extensible, the initial implementation has to understand a particular subset of attributes. The following list the attributes and understood, and any special coding conditions for that given attribute.
While the key and certificate database format is extensible, the initial implementation has to understand a particular subset of attributes. The following list of attribute types will be understood, and any special coding conditions for those attribute types.


*Stored in the key database:
*Stored in the key database:
Line 196: Line 224:
===== Special coding for encrypted entries =====
===== Special coding for encrypted entries =====


Encrypted entries are stored in the database as pkcs5 encoded blobs.
Encrypted entries are stored in the database as PKCS5 encoded blobs.
  SEQUENCE {
  SEQUENCE {
   AlgorithmID algorithm,
   AlgorithmID algorithm,
Line 338: Line 366:
==== User App Initialization and System App Initialization ====
==== User App Initialization and System App Initialization ====


One of the goals of making a shared database version of NSS is to create a 'system crypto library' in which applications will automatically share database and configuration settings. In order for this to work, applications need to be able to open NSS databases from standard locations.
One of the goals of making a shareable database version of NSS is to create a 'system crypto library' in which applications will automatically share database and configuration settings. In order for this to work, applications need to be able to open NSS databases from standard locations.


This design assumes that new NSS init functions will be defined for applications wanting to do 'standard user initialization', rather than building special knowledge into softoken or the database model. Note: This is different from the 2001 design, or and earlier prototype shared database, where the database code knew the location of the shared database.
This design assumes that new NSS init functions will be defined for applications wanting to do 'standard user initialization', rather than building special knowledge into softoken or the database model. Note: This is different from the 2001 design, or and earlier prototype shared database, where the database code knew the location of the shared database.
Line 344: Line 372:
==== Database Upgrade ====
==== Database Upgrade ====


Because there are some semantic differences between the Shared Database version
To understand the issues of migration to the Shareable Database version of NSS from the traditional (legacy) versions, we group applications that use the new version of NSS into three 'modes' of operation, and into two types, of which there are five valid combinations.  
of NSS and the traditional database, We really have 3 modes of operation for
the NSS and shared databases.


Mode 1: Legacy applications which upgrade to the new version of NSS without
Mode 1: Legacy applications which formerly used DBM databases, that upgrade to the new version of NSS without making any changes to the applications' code.
making changes to the application itself.


These applications will continue to use the legacy database support and the
These applications will continue to use the legacy database support and the
old database format. The applications cannot take advantage of new features
old database format. The applications cannot take advantage of new features
in the shared database. In this Mode, the nssdbm3 shared library must be  
in the shared database. In this Mode, the nssdbm3 shared library must be  
present. No update is needed in this case.
present. No update from legacy DBM to sharable is needed in this mode.


   Summary:
   Summary:
Line 361: Line 386:
   Application Changes: none.
   Application Changes: none.


Mode 2: Applications which maintain private copies of their certs and key  
Mode 2: Applications that use the new shareable database engine, but choose not to share copies of their cert and key stores.  They wish to keep separate copies of their databases.  They may or may not have existing legacy DBM databases from older versions of those applications. (Some servers might be like this.)
stores (various servers, for instance).


These applications will use the shared database. NSS will automatically  
These applications use the new shareable database engine. The first time the new application version runs, when NSS first creates the shareable databases, NSS will automatically detect instances of legacy databases and will upgrade those legacy databases to the new shareable ones without user interaction.
detect instances of legacy databases when first creating the shared database
and upgrade those legacy databases without user interaction.


   Summary:
   Summary:
Line 376: Line 398:
         trust.
         trust.


Mode 3: Applications which intend to share their keys and certs with other  
Mode 3: Applications that intend to share their keys and certs with other applications (the common case - browsers, mail clients, secure shells, vpns, etc.)
applications (the common case - browsers, mail clients, secure shells,  
vpns, etc.)


These applications will share a common database. In order for this to work,
To achieve that sharing, these applications must share a single common set of databases. If older versions of these applications created legacy DBM databases, those legacy databases must be merged. To perform such a merge, NSS will need some extra support from the application, and possibly user intervention as well.  
these applications must all open the same set of databases. Currently these
applications have their own independent copies of the databases. These
independent copies must be merged. In order or NSS to complete this merge, it
will need some extra support from the application itself, possibly user  
intervention as well.  


   Summary:
   Summary:
Line 397: Line 412:
     Changes to aid in update
     Changes to aid in update


Type A: Type A applications have existing NSS databases. These applications  
Type A: Type A applications are new versions of applications that existed before NSS supported sharable databases.  They have existing legacy NSS databases. The new versions of these applications have been upgraded to the new NSS that supports shareable databases.  If they intend to share the contents of those old databases, they need to merge the old database contents into the new ones.  
have upgraded to NSS, and need to merge old databases. All Mode 1 applications
 
are type A. All Type A applications need nssdbm3 to either upgrade or run.
All Mode 1 applications are type A and need nssdbm3 at all times. Mode 2 and Mode 3 applications of Type A need nssdbm3 to upgrade from the old legacy DBM databases to the new shareable databases.  Mode 3 Type A applications need libnssdbm3 to merge data from legacy DBM databases into shareable ones.  Mode 2 and Mode 3 type A applications do not need nssdbm3 except for upgrading and merging data from old legacy databases.


Type B: Type B applications are applications ported to NSS after the new shared
Type B: Type B applications are applications that never used any old version of NSS that supported only DBM databases.  All NSS databases used by Type B applications are sharable databases. There are no legacy DBM databases for Type B applications. All Type B applications are either Mode 2 or Mode 3.  
database code. All Type B applications should be either Mode 2 or Mode 3.  
Type B applications do no database upgrades, and do not need nssdbm3.
Type B applications need no update interactions, nor do the need nssdbm3.




=====Update complications=====
=====Upgrade complications=====


Updated complications only affect Type A applications. In order to merge a  
Upgrade complications only affect Type A applications. In order to merge a  
legacy database into an existing shared database, NSS needs the password for  
legacy database into an existing shareable database, NSS needs the password for  
both databases.  
both databases.  


Line 479: Line 493:


In Mode 3, the new database may or may not be initialized. For the first mode 3
In Mode 3, the new database may or may not be initialized. For the first mode 3
application, the new database will be uninitialized. NSS can procede the with
application, the new database will be uninitialized. NSS can proceed the with
the same procedure as Mode 2. When the second and subsequent applications  
the same procedure as Mode 2. When the second and subsequent applications  
start, the new shared database will already be initialized with it's own
start, the new shared database will already be initialized with it's own
Nelsonb
106

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/73917"