NSS Shared DB

Shared Database Proposal

Applications have been chaffing at the restrictions of the current NSS database for quite some time now. In 2001 we built some tools to work around those restrictions so certain applications could share the database if they supplied their own shared database implementation, and configured NSS to use that implementation. Today we have a process level, ACID, open source, and widely available database called SQLite. In addition, there is a strong desire to make NSS the system security service for Linux. I am proposing how we could leverage this database to give all of our applications Shared Database access.

Where we are today

At initialization time, NSS currently takes an argument which points to some directory the application uses to store it's private configuration data. NSS uses 3 libdbm files in that directory:

• cert8.db - stores publicly accessible objects (certs, crls, smime records).
• key3.db - stores the private keys.
• secmod.db - stores the pkcs #11 module configuration.

In addition:

• NSS may also use a directory called cert8.dir to store very large blobs (typically large CRLs).
• NSS may read from from previous certificate database (cert7.db, cert5.db, etc.) and build a new cert8.db if it doesn't exist.

These files are all accessed through the softoken, making libsoftokn3.so the only NSS library that needs to link with libdbm.

If the directory argument passed to NSS starts with the string 'multiaccess:', NSS does not use it as a directory path. Instead, NSS parses the string out as follows:

multiaccess:appName[:directory]

Where:
multicaccess is a keyword.
appName is a unique string for each group of applications which share the database.
directory is an optional parameter pointing to an NSS non-shared database which NSS will use to update the shared database from on loading.

NSS will find librdb.so (rdb.dll on windows) in it's path and load it. This shared library is expected to implement a superset of the standard libdbm interface. The main entry point is rdbopen, which will pass the appName, database name, and open flags. The rdb shared library will pick a location or method to store the database (it may not necessarily be a file), then handle the raw db records form NSS. The library does not do any formatting of the data.

What we want to do

We want to move key3 and cert8 into a new SQL databases called key4.db and cert9.db. This database will store the PKCS #11 objects currently stored or implied in cert8 and key3.

Optionally the databases could be combined into a single database, cert9.db, where private and public objects are stored in separate tables.

Review input required: is there a preference between a single DB file or separate key and cert DB files?

schema

The schema for the database will be simple.

• Each row will represent an object.
• The row schema will contain the the Object ID and the list of known Attribute Types.
• Newer versions of NSS may add new attribute types on the fly as necessary (extending the schema).
• The Attribute values will be stored as binary blobs.
  • Attributes that represent CK_ULONG values will be stored as 32-bit values in network byte order.
  • All other objects, byte order is already specified by PKCS #11.
  • Private attributes will be encrypted with a pkcs5 PBE in the same way private and secret keys are encrypted today.
  • Softoken will only set those attributes appropriate for the given object. If The attribute is not appropriate it will be left blank. (Note that sqlite does not distinguish between a NULL attribute and an empty one. This can be handled in the following ways:
    1. add a 'record' value around each attribute, where meta-data can be stored in the record. one of the meta-data values is that the record is empty.
    2. don't try to distinguish between a NULL attribute and a non-existent one.
    3. store a special value which means 'NULL' when writing a NULL record. (This is the current proposal).

Out of band data necessary for the proper operation of softoken, but not reflected back to NSS in general (currently this only applies to password entries), will be stored in separate tables in the database with their own schema.

Database extension is accomplished in 2 ways:

1. New attributes are added to the list known attributes and to already defined PKCS #11 objects. Older database objects can be detected because they will have 'invalid' values for these attributes (example, could add CKA_ONLY_FOR_URL to trust objects).
2. Add new PKCS #11 objects to hold the data (example, could add a new SSL_DATA record to store mappings to various certificates to different cipher suites and host name*)

NOTE: * I'm not suggesting this be the design for handling the SSL mapping problem, only as an example for the kind of thing that can be added to the database even after this deployment.

Accessing the shared Database

In order to maintain binary compatibility, I propose extending the keyword usage in NSS as follows:

multiaccess:appName[:directory] works as it does today, including using the cert8/key3 record version.
dbm:directory opens an existing non-shared libdbm version 8 database.
sql:directory1[:directory2] opens a shared database, cert9.db (& key4.db) in directory1 if cert9.db does exist. If the database does not exist, then directory2 is searched for a libdbm cert8.db and key3.db. If directory2 is not supplied, directory1 is searched.
extern:directory open a sql-like database by loading an external module, a. la. rdb and multiaccess:. This option would not be implemented in the initial release, but the extern: keyword would be reserved for future use.

Plain directory spec. For binary compatibility, we can treat the plain directory spec as the same as dbm:directory. In this case applications will not need to change for this release of NSS. (particularly unfriendly applications that want to tweak with the actual database file). Alternatively, we could treat it as sql:directory. In this case existing applications will get the benefits of shared databases immediately, though we risk breaking those aforementioned applications.

Reviewer input required: which default would you prefer?

When accessing the dbm: and multiaccess: directories, external shared library will be loaded with knows how to handle these legacy databases. This allows us to move much of the current mapping code into this shared library.

Secmod.db

In the dbm: and multiaccess: cases, there will be no changes to secmod.db.

In the sql: case, a new flat text file of the format specified in the PKCS #11 working group, but not yet included in any spec. This file will be opened, locked, used, then closed (much like the current secmod.db). The file will live in the same directory as cert9.db/key4.db. As a flat file it would not use the sqlite database to access these records. The file name should become pkcs11.txt or secmod.txt

User App Initialization and System App Initialization

One of the goals of making a shared database version of NSS is to create a 'system crypto library' in which applications will automatically share database and configuration settings. In order for this to work, applications need to be able to open NSS databases from standard locations.

This design assumes that new NSS init functions will be defined for applications wanting to do 'standard user initialization', rather than building special knowledge into softoken or the database model. Note: This is different from the 2001 design, or and earlier prototype shared database, where the database code knew the location of the shared database.

Other issues

Review Input Requested: Question, should we 'mark' old cert8/key3 databases as having been used to upgrade the shared database?

The issue here is applications that were once separated, but now combined. It would be nice if the keys and certs in each of these databases were properly merged into the overall shared database.

Upgrade itself is an issue. Do we want to support automatic upgrade (as we do today for older cert db types), or do we want to supply the application with an upgrade tool. The complication comes when we need to upgrade multiple old databases into a new one.

Layering

In order to keep clean separation between the data and database operations, we will continue to maintain an layer between the actual data handling and interpretation and the database itself. The database code will not need to understand:

1. What objects are actually stored in it.
2. The types of the attributes.
3. The meaning of the stored data.

Softoken (not the database) will manage cannonicalizing any CK_ULONGS, encrypting or decrypting private data blobs, and deciding what attributes an object should have and setting the appropriate defaults if necessary.

Since softoken deals with PKCS #11 templates internally, It's interface to the database will be in terms of those templates.

The database layer must be multi-thread safe. If the underlying database is not thread safe, sdb_ layer must implement the appropriate locking.

s_open

The database API consists of an initialization call, which returns an SDB data structure (defined below).

CK_RV
s_open(char *directory, int cert_version,
       int key_version, int flags, SDB **certdb, SDB **keydb)

The sdb_init function takes:

• directory full path to where the database lives.
• cert_version current version is the current database version
• key_version is the current key database version
• flags are:
  • FORCE
  • READONLY
  • READ/WRITE/CREATE
• certdb is the returned cert SDB structure
• keydb is the returned key SDB structure

(NOTE: the signature of this function is likely to change. Comments on how to change it would be appreciated.

Issues include:
1. Should cert and key versions be part of the API, or should they be part of the underlying database? 2. The cert and key prefix values are missing and should be added. 3. Do we need separate create flag or is the combined READ/WRITE/CREATE sufficient? 4. need a way to supply the list of know attributes so old DB's can be updated and initialized properly?

The returned SDB structure has the following format:

typedef struct SDBStr SDB;

struct SDBStr {
   void *private;
   int  sdb_type;
   int  sdb_flags;
   int  sdb_version;
   CK_RV (*sdb_FindObjectsInit)(SDB *sdb, const CK_ATTRIBUTE *template,
                                int count, SDBFind **find);
   CK_RV (*sdb_FindObjects)(SDB *sdb, SDBFind *find, CK_OBJECT_HANDLE *ids,
                               int arraySize, int *count);
   CK_RV (*sdb_FindObjectsFinal)(SDB *sdb, SDBFind *find);
   CK_RV (*sdb_GetAttributeValue)(SDB *sdb, CK_OBJECT_HANDLE object,
                               CK_ATTRIBUTE *template, int count);
   CK_RV (*sdb_SetAttributeValue)(SDB *sdb, CK_OBJECT_HANDLE object,
                               const CK_ATTRIBUTE *template, int count);
   CK_RV (*sdb_CreateObject)(SDB *sdb, CK_OBJECT_HANDLE *object,
                               const CK_ATTRIBUTE *template, int count);
   CK_RV (*sdb_DestroyObject)(SDB *sdb, CK_OBJECT_HANDLE object);
   CK_RV (*sdb_GetPWEntry)(SDB *sdb, SDBPasswordEntry *entry);
   CK_RV (*sdb_PutPWEntry)(SDB *sdb, SDBPasswordEntry *entry);
   CK_RV (*sdb_Begin)(SDB *sdb);
   CK_RV (*sdb_Commit)(SDB *sdb);
   CK_RV (*sdb_Abort)(SDB *sdb);
   CK_RV (*sdb_Close)(SDB *sdb);
};

where:

• private is a pointer to opaque private data specific to the Shared DB implementation.
• sdb_type is the type of database (key [aka private] or cert [aka public]).
• sdb_flags specifies how the database was opened (ReadOnly, Create, etc).
• sdb_version specifies the version of the underlying sdb structure. This allows us to handle future expansion of the sdb data structure safely.
• The rest are function pointers to database primitives described next.

sdb_FindObjectsInit

CK_RV (*sdb_FindObjectsInit)(SDB *sdb, const CK_ATTRIBUTE *template,
                                int count, SDBFind **find);

This function is the equivalent of PKCS #11 C_FindObjectsInit(). It returns a SDBFind context with is opaque to the caller. The caller must call sdb_FindObjectsFinal with this context if sdb_FindobjectsInit succeeds.

sdb_FindObjects

CK_RV (*sdb_FindObjects)(SDB *sdb, SDBFind *find, CK_OBJECT_HANDLE *ids,
                               int arraySize, int *count);

This function is the equivalent of PKCS #11 C_FindObjects(). It takes a SDBFind context returned by sdb_FindObjectsInit. This function has the same semantics as C_FindObjects with respect to handling how many objects are returned in a single call.

sdb_FindObjectsFinal

CK_RV (*sdb_FindObjectsFinal)(SDB *sdb, SDBFind *find);

This function is the equivalent of PKCS #11 C_FindObjectsFinal(). It frees any resources associated with SDBFIND.

sdb_GetAttributeValue

CK_RV (*sdb_GetAttributeValue)(SDB *sdb, CK_OBJECT_HANDLE object,
                               CK_ATTRIBUTE *template, int count);

This function is the equivalent of PKCS #11 C_GetAttributeValue(). It has the same memory allocation and error code semantics of the PKCS #11 call. The attributes passed to sdb_GetAttributeValues are already transformed from their native representations in the following ways:

1. CKU_LONG values are stored as 32-bit values in network byte order.
2. Private attributes will be encrypted.

sdb_SetAttributeValue

CK_RV (*sdb_SetAttributeValue)(SDB *sdb, CK_OBJECT_HANDLE object,
                               const CK_ATTRIBUTE *template, int count);

This function is the equivalent of PKCS #11 C_SetAttributeValue(). The attributes returned to sdb_SetAttributeValues are transformed from their native representations in the following ways:

1. CKU_LONG values returned 32-bit values in network byte order.
2. Private attributes returned encrypted.

sdb_CreateObject

CK_RV (*sdb_CreateObject)(SDB *sdb, CK_OBJECT_HANDLE *object,
                               const CK_ATTRIBUTE *template, int count);

This function is the equivalent of PKCS #11 C_CreateObject(). The value of 'object' is chosen by the implementer of sdb_CreateObject. This value must be unique for this sdb instance. It should be no more than 30 bits long.

sdb_DestroyObject

CK_RV (*sdb_DestroyObject)(SDB *sdb, CK_OBJECT_HANDLE object);

This function is the equivalent of PKCS #11 C_Destroy object(). It removed the object from the database.

sdb_GetPWEntry

CK_RV (*sdb_GetPWEntry)(SDB *sdb, SDBPasswordEntry *entry);

Get the password entry. This only applies to the private database.

Review input needed: Would it be better to define a 'metadata' operation where we call the database to fetch data that is not reflected through the PKCS #11 interface?

sdb_PutPWEntry

CK_RV (*sdb_PutPWEntry)(SDB *sdb, SDBPasswordEntry *entry);

Write the password entry. This only applies to the private database. Writing a password entry will overwrite the old entry.

Would it be better to define a 'metadata' operation where we call the database to put data that is not reflected through the PKCS #11 interface?

sdb_Begin

CK_RV (*sdb_Begin)(SDB *sdb);

Begins a transaction. Any write to the database (sdb_CreateObject, sdb_DestroyObject, sdb_SetAttributeValue) must be accomplished while holding a transaction. Transactions are completed by calling sdb_Commit to commit the change, and sdb_Abort to discard the change. More than one write operation may be made while holding a transaction. Aborting the transaction will discard all writes made while in the transaction.

sdb_Commit

CK_RV (*sdb_Commit)(SDB *sdb);

Commit a transaction. Any write to the database (sdb_CreateObject, sdb_DestroyObject, sdb_SetAttributeValue) must be accomplished while holding a transaction. Transactions are completed by calling sdb_Commit to commit the change, and sdb_Abort to discard the change. More than one write operation may be made while holding a transaction.

sdb_Abort

CK_RV (*sdb_Abort)(SDB *sdb);

Abort a transaction. Any write to the database (sdb_CreateObject, sdb_DestroyObject, sdb_SetAttributeValue) must be accomplished while holding a transaction. Transactions are completed by calling sdb_Commit to commit the change, and sdb_Abort to discard the change. More than one write operation may be made while holding a transaction. Aborting the transaction will discard all writes made while in the transaction.

sdb_Close

CK_RV (*sdb_Close)(SDB *sdb);

Close the SDB and free up any resources associated with it.

legacy DB support

The old dbm code can be supported with the above SDB structure with the following exceptions:

1. The old db code cannot be extensible (can't dynamically handle new types).
2. A private interface may be needed to unwrap the private keys, or provide a handle to the password so the keys can be presented in the attribute format.

This code would live in it's own shared library. Most of the low level cert, crl, key handling, and translation to pkcs #11 objects and attributes that was part of softoken will moved to this legacy shared library. It will be loaded whenever access to old databases is required.