From Fedora Project Wiki
(initial copy)
 
(add section on slow network filesystems and the recommended environment variable NSS_SDB_USE_CACHE=yes)
 
(20 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{admon/important | Comments and Explanations | The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To read it, choose the "view source" link.<br/> '''Copy the source to a ''new page'' before making changes!  DO NOT EDIT THIS TEMPLATE FOR YOUR CHANGE PROPOSAL.'''}}
= NSS Default File Format SQL =
 
<!-- Self Contained or System Wide Change Proposal?
Use this guide to determine to which category your proposed change belongs to.
 
Self Contained Changes are:
* changes to isolated/leaf package without the impact on other packages/rest of the distribution
* limited scope changes without the impact on other packages/rest of the distribution
* coordinated effort within SIG with limited impact outside SIG functional area, accepted by the SIG
 
System Wide Changes are:
* changes that does not fit Self Contained Changes category touching
* changes that require coordination within the distribution (for example mass rebuilds, release engineering or other teams effort etc.)
* changing system defaults
 
For Self Contained Changes, sections marked as "REQUIRED FOR SYSTEM WIDE CHANGES" are OPTIONAL but FESCo/Wrangler can request more details (especially in case the change proposal category is
improper or updated to System Wide category). For System Wide Changes all fields on this form are required for FESCo acceptance (when applies). 
 
We request that you maintain the same order of sections so that all of the change proposal pages are uniform.
-->
 
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
 
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
= Change Proposal Name <!-- The name of your change proposal --> =


== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
Change the NSS library default to use the sqlite based data storage, when applications don't specify their preferred storage file format.


== Owner ==
== Owner ==
<!--
* Name: [[User:kengert| Kai Engert]]
For change proposals to qualify as self-contained, owners of all affected packages need to be included here. Alternatively, a SIG can be listed as an owner if it owns all affected packages.
* Email: kaie@redhat.com
This should link to your home wiki page so we know who you are.
* Release notes ticket: [https://pagure.io/fedora-docs/release-notes/issue/117 #117]
-->
* Name: [[User:FASAcountName| Your Name]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: <your email address so we can contact you, invite you to meetings, etc.>
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
Line 46: Line 17:


== Current status ==
== Current status ==
* Targeted release: [[Releases/<number> | Fedora <number> ]]  
* Targeted release: [[Releases/28 | Fedora 28 ]]  
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
Line 56: Line 27:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1474771 #1474771]


== Detailed Description ==
== Detailed Description ==
Applications that use the NSS library often use a database for storage of keys, certificates and trust. NSS supports two different file formats, one called DBM (based on berkeley DB files) and another one called SQL (based on sqlite DB files).


<!-- Expand on the summary, if appropriate. A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
Today's default file format used by NSS, used when applications omit the type parameter, is the older DBM file format, which forbids parallel access to the storage.
The suggestion is to change the default file format to SQL, which allows parallel access to the storage.
 
Applications, or users using the NSS command line utilities, often provide the database storage location using a simple directory path parameter. Some might not be aware, or forget, that the parameter can be prefixed with a type modifier, either "dbm:" or "sql:".
 
As a result, when not providing this parameter, the file format used will be the fragile DBM file format.
This is particuarly problematic, if a user attempts to modify the NSS storage using command line tools, while another process, such as a daemon, is running concurrently, which also accesses the same database in the DBM file format. This often results in corrupted database storage, which cannot be recovered.
 
By changing the default, all applications that currently use the DBM file format, will automatically be migrated to the SQL file format. NSS has the ability to discover if a storage location (a directory) contains the DBM file format. If configured to use the modern SQL format, NSS will automatically perform a one-time conversion from the DBM to the SQL format.
 
The same applies to the NSS command line utilities. If the NSS library default is changed to SQL, the NSS tools will also trigger the one-time conversion, or access the already converted files.


== Benefit to Fedora ==
== Benefit to Fedora ==
The benefit will be less data corruption, because the SQL file format explicitly allows parallel access to the storage files by multiple applications.


NSS has been supporting this new file format for many years, and several applications have already opted in to explicitly use the new file format, so it can be considered stable.
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners:
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
A small downstream patch needs to be applied to the NSS library package, which changes the library default.
 
* Other developers:
It's up to developers of NSS applications, if they accept the new default and an automatic conversion, or if they prefer to continue to use the classic DBM storage format. Although not recommended, developers can easily do so, by adding a "dbm:" prefix to the storage parameter they provide to NSS at NSS library initialization time.


* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Release engineering: https://pagure.io/releng/issue/6883
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
No help should be necessary. No mass rebuild necessary.


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: all
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->


* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Policies and guidelines: N/A
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->


* Trademark approval: N/A (not needed for this Change)
* Trademark approval: N/A
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


== Upgrade/compatibility impact ==
== Upgrade/compatibility impact ==
<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
Data will be migrated automatically on first access with the new default, if the new access triggers a code path that results in writing data.
 
Note that migration might happen in multiple steps. An application access, which doesn't provide the password for the key database, might trigger a migration of the contents of the cert8.db file to the cert9.db file, and might create a half-migrated key4.db file. On a future database access with the database password being available (application authenticates to the NSS database), NSS will complete the migration of the key3.db contents to the key4.db file.
 
In some scenarios it might be preferable to trigger an explicit migration of all contents. For example, if a daemon usually runs with read-only access to a database, and selinux restricts the daemon from writing the NSS database files, an automatic migration attempt may fail to write the migrated database files.
 
The following command can be used to trigger an explicit migration:
 
certutil -d sql:/path/to/database -N -f /path/to/database/password/file -@ /path/to/database/password/file


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
Although the -N command is primarily intended for creating a new database, it can also be used for migration inside a database directory. If an NSS database is already present in the directory provided with the -d parameter, no new databases will be created. In other words, the -N command is safe to use on existing database directories, it will not erase.
N/A (not a System Wide Change)  
 
To create the correct parameter for the -d option, concat "sql:" and your /path/to/database, don't use space characters between the prefix and the path.
 
The -f and -@ parameters can be used to avoid interactive prompts during the migration. If you prefer to, you may run the command without these parameters, and enter your database password interactively three times. (For databases that aren't protected by a password, you may press enter at each of the three prompts for old and new password.)
 
If you'd like to perform a migration that doesn't prompt interactively, prepare a text file that contains the database password. (If the database uses an empty password, the text file should contain one empty line, or in other words, a newline character). For both -f and -@ parameters, provide the path to the password file.
 
The migration will not remove the old database files key3.db, cert8.db and secmod.db. You may decide to keep them for backup purposes. If you are using a NSS library that has been changed to use the sql database format by default, or if the application explicity uses the sql: prefix for the NSS database parameter, then NSS will use the new files key4.db, cert9.db and pkcs11.txt, only.
 
(See also https://bugzilla.mozilla.org/show_bug.cgi?id=1415912 which contains discussion about the recommended migration approach.)
 
The migration will not preserve file ownership and file permission of the old dbm database files, when creating the new database files. If special ownership or permissions are required, the explicit migration should be followed by appropriate calls to chmod / chown commands.


== How To Test ==
== How To Test ==
<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done. If it needs to be tested with different hardware or software configurations, indicate them. The more specific you can be, the better the community testing can be.  
Each application or server softare that uses NSS should be tested for successful data migration. Ideally, an application/server configuration should be used, which already contains data from a previous Fedora version. It should be tested that after migration to the new Fedora version, the software continues to operate as expected. In particular, this means that the software still has access to the locally configure private keys, certificates, trust settings which the application/server has configured. (This only applies to applications that perform this kind of operation.)
 
== User Experience ==
The result changes will be restricted to the file system level, and to applications that use the NSS library default.


Remember that you are writing this how to for interested testers to use to check out your change implementation - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your change.
Previously, the following filenames were used for NSS storage:
* cert8.db
* key3.db
* secmod.db


A good "how to test" should answer these four questions:
After the change, the following filenames will be used:
* cert9.db
* key4.db
* pkcs11.txt


0. What special hardware / data / etc. is needed (if any)?
Applications that migrate data from an older version will contain both sets of files in their storage. However, after the conversion, the old files will no longer be used, they can be considered a one-time backup.
1. How do I prepare my system to test this change? What packages
 
need to be installed, config files edited, etc.?
Applications that are executed for the first time with the new library default already effective, will only get the new storage files.
2. What specific actions do I perform to check that the change is
 
working like it's supposed to?
== Performance with Network Filesystems (NFS, Samba) ==
3. What are the expected results of those actions?
Users of network filesystems may experience a major slowdown with some applications, such as Firefox.
-->


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
When storing a user's home or application directory on a network filesystem, it is recommended to set the environment variable NSS_SDB_USE_CACHE=yes prior to starting applications that use NSS.
N/A (not a System Wide Change)


== User Experience ==
The environment variable enables the use of a caching strategy in NSS, that works around the slowness of network filesystems. Because this caching strategy causes a performance decrease on fast filesystems, it isn't used by default, but requires users to explicitly enable it.
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)


== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
No dependencies.
 
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)


== Contingency Plan ==
== Contingency Plan ==
 
* Contingency mechanism: (What to do?  Who will do it?) Revert the shipped configuration, NSS maintainer.
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency deadline: beta freeze
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Blocks product? No
* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next -->


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
There are couple of upstream documents, which have been produced a couple of years ago, which might provide background on this change of technology.
* https://wiki.mozilla.org/NSS_Shared_DB
* https://wiki.mozilla.org/NSS_Shared_DB_And_LINUX


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
(Those documents partially focused on an additional property, which could potentially get achieved by migrating to the new file format: Having a common database for each system user, shared by all applications one user uses.
N/A (not a System Wide Change)  
However, this potential consolidation is more complicated to achieve. I suggest to EXCLUDE the consolidation from this change. The reason is the unresolved issue of merging databases from two sources to a single destination, when the sources used different database protection passwords. Migration will fail, unless application provides a user interface to prompt for the second database password.)


== Release Notes ==
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
Yes, we should provide detailed release notes.
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this change, indicate them here.  A link to upstream documentation will often satisfy this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release.
 
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
-->


[[Category:ChangePageIncomplete]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->


<!-- Select proper category, default is Self Contained Change -->
[[Category:ChangeAcceptedF28]]
[[Category:SelfContainedChange]]
[[Category:SystemWideChange]]
<!-- [[Category:SystemWideChange]] -->

Latest revision as of 12:17, 27 March 2018

NSS Default File Format SQL

Summary

Change the NSS library default to use the sqlite based data storage, when applications don't specify their preferred storage file format.

Owner

Current status

Detailed Description

Applications that use the NSS library often use a database for storage of keys, certificates and trust. NSS supports two different file formats, one called DBM (based on berkeley DB files) and another one called SQL (based on sqlite DB files).

Today's default file format used by NSS, used when applications omit the type parameter, is the older DBM file format, which forbids parallel access to the storage. The suggestion is to change the default file format to SQL, which allows parallel access to the storage.

Applications, or users using the NSS command line utilities, often provide the database storage location using a simple directory path parameter. Some might not be aware, or forget, that the parameter can be prefixed with a type modifier, either "dbm:" or "sql:".

As a result, when not providing this parameter, the file format used will be the fragile DBM file format. This is particuarly problematic, if a user attempts to modify the NSS storage using command line tools, while another process, such as a daemon, is running concurrently, which also accesses the same database in the DBM file format. This often results in corrupted database storage, which cannot be recovered.

By changing the default, all applications that currently use the DBM file format, will automatically be migrated to the SQL file format. NSS has the ability to discover if a storage location (a directory) contains the DBM file format. If configured to use the modern SQL format, NSS will automatically perform a one-time conversion from the DBM to the SQL format.

The same applies to the NSS command line utilities. If the NSS library default is changed to SQL, the NSS tools will also trigger the one-time conversion, or access the already converted files.

Benefit to Fedora

The benefit will be less data corruption, because the SQL file format explicitly allows parallel access to the storage files by multiple applications.

NSS has been supporting this new file format for many years, and several applications have already opted in to explicitly use the new file format, so it can be considered stable.

Scope

  • Proposal owners:

A small downstream patch needs to be applied to the NSS library package, which changes the library default.

  • Other developers:

It's up to developers of NSS applications, if they accept the new default and an automatic conversion, or if they prefer to continue to use the classic DBM storage format. Although not recommended, developers can easily do so, by adding a "dbm:" prefix to the storage parameter they provide to NSS at NSS library initialization time.

No help should be necessary. No mass rebuild necessary.

  • Policies and guidelines: N/A
  • Trademark approval: N/A

Upgrade/compatibility impact

Data will be migrated automatically on first access with the new default, if the new access triggers a code path that results in writing data.

Note that migration might happen in multiple steps. An application access, which doesn't provide the password for the key database, might trigger a migration of the contents of the cert8.db file to the cert9.db file, and might create a half-migrated key4.db file. On a future database access with the database password being available (application authenticates to the NSS database), NSS will complete the migration of the key3.db contents to the key4.db file.

In some scenarios it might be preferable to trigger an explicit migration of all contents. For example, if a daemon usually runs with read-only access to a database, and selinux restricts the daemon from writing the NSS database files, an automatic migration attempt may fail to write the migrated database files.

The following command can be used to trigger an explicit migration:

certutil -d sql:/path/to/database -N -f /path/to/database/password/file -@ /path/to/database/password/file

Although the -N command is primarily intended for creating a new database, it can also be used for migration inside a database directory. If an NSS database is already present in the directory provided with the -d parameter, no new databases will be created. In other words, the -N command is safe to use on existing database directories, it will not erase.

To create the correct parameter for the -d option, concat "sql:" and your /path/to/database, don't use space characters between the prefix and the path.

The -f and -@ parameters can be used to avoid interactive prompts during the migration. If you prefer to, you may run the command without these parameters, and enter your database password interactively three times. (For databases that aren't protected by a password, you may press enter at each of the three prompts for old and new password.)

If you'd like to perform a migration that doesn't prompt interactively, prepare a text file that contains the database password. (If the database uses an empty password, the text file should contain one empty line, or in other words, a newline character). For both -f and -@ parameters, provide the path to the password file.

The migration will not remove the old database files key3.db, cert8.db and secmod.db. You may decide to keep them for backup purposes. If you are using a NSS library that has been changed to use the sql database format by default, or if the application explicity uses the sql: prefix for the NSS database parameter, then NSS will use the new files key4.db, cert9.db and pkcs11.txt, only.

(See also https://bugzilla.mozilla.org/show_bug.cgi?id=1415912 which contains discussion about the recommended migration approach.)

The migration will not preserve file ownership and file permission of the old dbm database files, when creating the new database files. If special ownership or permissions are required, the explicit migration should be followed by appropriate calls to chmod / chown commands.

How To Test

Each application or server softare that uses NSS should be tested for successful data migration. Ideally, an application/server configuration should be used, which already contains data from a previous Fedora version. It should be tested that after migration to the new Fedora version, the software continues to operate as expected. In particular, this means that the software still has access to the locally configure private keys, certificates, trust settings which the application/server has configured. (This only applies to applications that perform this kind of operation.)

User Experience

The result changes will be restricted to the file system level, and to applications that use the NSS library default.

Previously, the following filenames were used for NSS storage:

  • cert8.db
  • key3.db
  • secmod.db

After the change, the following filenames will be used:

  • cert9.db
  • key4.db
  • pkcs11.txt

Applications that migrate data from an older version will contain both sets of files in their storage. However, after the conversion, the old files will no longer be used, they can be considered a one-time backup.

Applications that are executed for the first time with the new library default already effective, will only get the new storage files.

Performance with Network Filesystems (NFS, Samba)

Users of network filesystems may experience a major slowdown with some applications, such as Firefox.

When storing a user's home or application directory on a network filesystem, it is recommended to set the environment variable NSS_SDB_USE_CACHE=yes prior to starting applications that use NSS.

The environment variable enables the use of a caching strategy in NSS, that works around the slowness of network filesystems. Because this caching strategy causes a performance decrease on fast filesystems, it isn't used by default, but requires users to explicitly enable it.

Dependencies

No dependencies.

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) Revert the shipped configuration, NSS maintainer.
  • Contingency deadline: beta freeze
  • Blocks release? No
  • Blocks product? No

Documentation

There are couple of upstream documents, which have been produced a couple of years ago, which might provide background on this change of technology.

(Those documents partially focused on an additional property, which could potentially get achieved by migrating to the new file format: Having a common database for each system user, shared by all applications one user uses. However, this potential consolidation is more complicated to achieve. I suggest to EXCLUDE the consolidation from this change. The reason is the unresolved issue of merging databases from two sources to a single destination, when the sources used different database protection passwords. Migration will fail, unless application provides a user interface to prompt for the second database password.)

Release Notes

Yes, we should provide detailed release notes.