Software Update:MAR: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
m (→‎SIGNATURE_ENTRY.SignatureAlgorithmID: Fix SHA384 signature size)
 
(43 intermediate revisions by 4 users not shown)
Line 6: Line 6:


The file structure in a nutshell is a header (HEADER), followed by a (SIGNATURES) block, followed by a list of variable length files, and finally ending with an index of the files (INDEX).  The index is a list of variable length entries (INDEX_ENTRY).  The signatures block is a list of variable length entries (SIGNATURE_ENTRY).
The file structure in a nutshell is a header (HEADER), followed by a (SIGNATURES) block, followed by a list of variable length files, and finally ending with an index of the files (INDEX).  The index is a list of variable length entries (INDEX_ENTRY).  The signatures block is a list of variable length entries (SIGNATURE_ENTRY).


HEADER
HEADER
   4 bytes : MARID - "MAR1"
   4 bytes : MARID - "MAR1"
   4 bytes : OffsetToIndex - offset to INDEX in bytes relative to the start of MAR file
   4 bytes : OffsetToIndex - offset to INDEX in bytes relative to the start of MAR file
  8 bytes : FileSize - size in bytes of the entire MAR file


SIGNATURES
SIGNATURES
  8 bytes : FileSize - size in bytes of the entire MAR file
   4 bytes : NumSignatures - Number of signatures
   4 bytes : NumSignatures - Number of signatures
   ''NumSignatures SIGNATURE_ENTRY elements''
   ''NumSignatures SIGNATURE_ENTRY elements''


SIGNATURE_ENTRY
SIGNATURE_ENTRY (Only present if SIGNATURES.NumSignatures is more than 0)
   4 bytes : SignatureAlgorithmID - ID representing the type of signature algorithm.
   4 bytes : SignatureAlgorithmID - ID representing the type of signature algorithm.
   4 bytes : SignatureSize - Size in bytes of the signature that follows
   4 bytes : SignatureSize - Size in bytes of the signature that follows
   N bytes : Signature - The signature of type SIGNATURE_ENTRY.SignatureAlgorithmID and size N = SIGNATURE_ENTRY.SignatureSize bytes
   N bytes : Signature - The signature of type <tt>SIGNATURE_ENTRY.SignatureAlgorithmID</tt> and size N = <tt>SIGNATURE_ENTRY.SignatureSize</tt> bytes
 
ADDITIONAL_SECTIONS (Optional) 
  4 bytes: NumAdditionalSections - Number of additional sections
  ''NumAdditionalSections ADDITIONAL_SECTION_ENTRY elements''
 
ADDITIONAL_SECTION_ENTRY (Only present if NumAdditionalSections is more than 0)
  4 bytes: BlockSize - (Z)
  4 bytes: BlockIdentifier - Used to identify the additional section, 1 for a Product Information Block.
  (Z-8) bytes: The rest of the block.
 
CONTENT
  Y bytes : content data interpreted as per the INDEX_ENTRY elements. Where Y is the summation of all INDEX_ENTRY.ContentSize values.


INDEX
INDEX
Line 33: Line 44:
   1 byte  : null terminator
   1 byte  : null terminator


Some old MAR files will not contain the SIGNATURE block nor the HEADER.FileSize field.  Old parsers simply skip over these fields because they ignore everything between the MAR header and the offset given in the <tt>HEADER.OffsetToIndex</tt> field.
Some old MAR files will not contain the SIGNATURE block.  Old parsers simply skip over these fields because they ignore everything between the MAR header and the offset given in the <tt>HEADER.OffsetToIndex</tt> field.


== SIGNATURE blocks ==
== SIGNATURE blocks ==
Line 44: Line 55:
   
   
   1: RSA-PKCS1-SHA1 (2048 bits / 256 bytes)
   1: RSA-PKCS1-SHA1 (2048 bits / 256 bytes)
  2: RSA-PKCS1-SHA384 (4096 bits / 512 bytes)


The updater will only accept the MAR file if at least one of the signatures verifies.
The updater will only accept the MAR file if at least one of the signatures verify.
Some versions of the updater may not apply a MAR file unless a valid signature of a particular <tt>SIGNATURE_ENTRY.SignatureAlgorithmID</tt> is included in the MAR file.
Some versions of the updater may not apply a MAR file unless a valid signature of a particular <tt>SIGNATURE_ENTRY.SignatureAlgorithmID</tt> is included in the MAR file.


As of Firefox 10, only RSA-PKCS1-SHA1 signatures are accepted.
From Firefox 10-56, only RSA-PKCS1-SHA1 signatures are accepted. [https://bugzilla.mozilla.org/show_bug.cgi?id=1105689 Bug 1105689] adds support for SHA-384 and is slated for Firefox 56.
There is no indicator in the MAR file for which operating system the MAR is for.
There is no indicator in the MAR file for which operating system the MAR is for.


Line 56: Line 68:
The signatures are in NSS / OpenSSL / big-endian order and not CryptoAPI order.
The signatures are in NSS / OpenSSL / big-endian order and not CryptoAPI order.
If CryptoAPI is used to check a signature, the bytes of the signature must be reversed before verifying the signature using CryptVerifySignature.
If CryptoAPI is used to check a signature, the bytes of the signature must be reversed before verifying the signature using CryptVerifySignature.
== Constraints ==
To protect against invalid inputs the following constraints are in place:
* There are at most 8 signatures.
* The file size of the MAR file is at most 500MB.
* No signature is more than 2048 bytes long.


== Additional sections ==
== Additional sections ==
Additional MAR file sections may be defined in the future. 
If at least one additional section exists, the ADDITIONAL_SECTIONS header entry must be present.
Each additional block must be in the format described above in ADDITIONAL_SECTION_ENTRY.
The Block identifier is a value used to uniquely identify additional blocks so that they may be optionally added in any order.


Additional MAR file sections may defined in the future. These sections must follow directly after the SIGNATURES section.
=== Product Information Block ===
 
The product information block identifies the product and channel this MAR file applies to.  It also includes the new version number to avoid downgrades.
The product and channel are combined in the MARChannelName field.
 
PRODUCT INFORMATION BLOCK
 
  4 bytes: BlockSize - The size of the product information block.
  4 bytes: BlockIdentifier - The identifier of the product information block, with a value of 1
  <64 bytes: MARChannelName used to identify the product and channel (such as from MAR_CHANNEL_ID) Example: mozilla-central
  1 byte: null terminator
  <32 bytes: ProductVersion string (such as from MOZ_APP_VERSION) Examples: 12.0.1.5371, 12.0a1
  1 byte: null terminator
  * bytes: Optional unused data adding up to BlockSize.


== Source Code ==
== Source Code ==
Line 69: Line 108:


This question was given a fair amount of consideration.  Ultimately, we decided to go with a custom file format because using libjar would have required a fair bit of hacking.  Writing custom code was a simpler option, and it resulted in less code (mar_read.c is less than 300 lines of code).  Moreover, the update system does not need a standard file format.  The elements stored in the archive are bzip2 compressed binary diffs, generated using a variation of bsdiff.  So, being able to unpack the archive file using standard tools wouldn't be very useful in and of itself.
This question was given a fair amount of consideration.  Ultimately, we decided to go with a custom file format because using libjar would have required a fair bit of hacking.  Writing custom code was a simpler option, and it resulted in less code (mar_read.c is less than 300 lines of code).  Moreover, the update system does not need a standard file format.  The elements stored in the archive are bzip2 compressed binary diffs, generated using a variation of bsdiff.  So, being able to unpack the archive file using standard tools wouldn't be very useful in and of itself.
== MAR extraction tools ==
There’s 2 different python scripts:
* http://hg.mozilla.org/build/tools/file/b77b517eab66/buildfarm/utils/mar.py
* https://github.com/mozilla/build-mar
Note:
* The latter is more recent, but has more dependencies.
* The former works just fine for extracting files
** to use the former: cd tmp_dir; mar.py -x ../foo.mar
** for bzip2'ed files such as b2g mar files:  try 'mar.py -x -j fota-flame-update.mar'
** or 'mv update.zip update.zip.bz2; bunzip2 update.zip.bz2'

Latest revision as of 14:20, 18 July 2018

Mozilla ARchive

This document describes the Mozilla ARchive (MAR) format used by the update system to deliver update packages. It is basically a series of files with an index tacked on to the end.

Details

The file structure in a nutshell is a header (HEADER), followed by a (SIGNATURES) block, followed by a list of variable length files, and finally ending with an index of the files (INDEX). The index is a list of variable length entries (INDEX_ENTRY). The signatures block is a list of variable length entries (SIGNATURE_ENTRY).

HEADER 

 4 bytes : MARID - "MAR1"
 4 bytes : OffsetToIndex - offset to INDEX in bytes relative to the start of MAR file

SIGNATURES 

 8 bytes : FileSize - size in bytes of the entire MAR file
 4 bytes : NumSignatures - Number of signatures
 NumSignatures SIGNATURE_ENTRY elements

SIGNATURE_ENTRY (Only present if SIGNATURES.NumSignatures is more than 0) 

 4 bytes : SignatureAlgorithmID - ID representing the type of signature algorithm.
 4 bytes : SignatureSize - Size in bytes of the signature that follows
 N bytes : Signature - The signature of type SIGNATURE_ENTRY.SignatureAlgorithmID and size N = SIGNATURE_ENTRY.SignatureSize bytes

ADDITIONAL_SECTIONS (Optional) 

 4 bytes: NumAdditionalSections - Number of additional sections
 NumAdditionalSections ADDITIONAL_SECTION_ENTRY elements

ADDITIONAL_SECTION_ENTRY (Only present if NumAdditionalSections is more than 0) 

 4 bytes: BlockSize - (Z)
 4 bytes: BlockIdentifier - Used to identify the additional section, 1 for a Product Information Block.
 (Z-8) bytes: The rest of the block.

CONTENT 

 Y bytes : content data interpreted as per the INDEX_ENTRY elements. Where Y is the summation of all INDEX_ENTRY.ContentSize values.

INDEX 

 4 bytes : IndexSize - Size of INDEX in bytes
 variable number of INDEX_ENTRY elements

INDEX_ENTRY 

 4 bytes : OffsetToContent - Offset in bytes relative to start of the MAR file
 4 bytes : ContentSize - Size in bytes of the content
 4 bytes : Flags - File permission bits (in standard unix-style format).
 M bytes : FileName - File name (byte array)
 1 byte  : null terminator

Some old MAR files will not contain the SIGNATURE block. Old parsers simply skip over these fields because they ignore everything between the MAR header and the offset given in the HEADER.OffsetToIndex field.

SIGNATURE blocks

Zero or more SIGNATURE_ENTRYs can be specified. The signatures must be composed of all bytes of the MAR file excluding the SIGNATURE_ENTRY.Signature fields. Each contained SIGNATURE_ENTRY.Signature must be of type SIGNATURE_ENTRY.SignatureAlgorithmID.

SIGNATURE_ENTRY.SignatureAlgorithmID

 1: RSA-PKCS1-SHA1 (2048 bits / 256 bytes)
 2: RSA-PKCS1-SHA384 (4096 bits / 512 bytes)

The updater will only accept the MAR file if at least one of the signatures verify. Some versions of the updater may not apply a MAR file unless a valid signature of a particular SIGNATURE_ENTRY.SignatureAlgorithmID is included in the MAR file.

From Firefox 10-56, only RSA-PKCS1-SHA1 signatures are accepted. Bug 1105689 adds support for SHA-384 and is slated for Firefox 56. There is no indicator in the MAR file for which operating system the MAR is for.

Byte Ordering

All fields are in big-endian format. The signatures are in NSS / OpenSSL / big-endian order and not CryptoAPI order. If CryptoAPI is used to check a signature, the bytes of the signature must be reversed before verifying the signature using CryptVerifySignature.

Constraints

To protect against invalid inputs the following constraints are in place:

  • There are at most 8 signatures.
  • The file size of the MAR file is at most 500MB.
  • No signature is more than 2048 bytes long.

Additional sections

Additional MAR file sections may be defined in the future. If at least one additional section exists, the ADDITIONAL_SECTIONS header entry must be present.

Each additional block must be in the format described above in ADDITIONAL_SECTION_ENTRY.

The Block identifier is a value used to uniquely identify additional blocks so that they may be optionally added in any order.

Product Information Block

The product information block identifies the product and channel this MAR file applies to. It also includes the new version number to avoid downgrades. The product and channel are combined in the MARChannelName field.

PRODUCT INFORMATION BLOCK 

 4 bytes: BlockSize - The size of the product information block.
 4 bytes: BlockIdentifier - The identifier of the product information block, with a value of 1
 <64 bytes: MARChannelName used to identify the product and channel (such as from MAR_CHANNEL_ID) Example: mozilla-central
 1 byte: null terminator
 <32 bytes: ProductVersion string (such as from MOZ_APP_VERSION) Examples: 12.0.1.5371, 12.0a1
 1 byte: null terminator
 * bytes: Optional unused data adding up to BlockSize.

Source Code

The source code can be found under modules/libmar.

See bug 296303 and bug 699700

Why not use ZIP or some other standard file format?

This question was given a fair amount of consideration. Ultimately, we decided to go with a custom file format because using libjar would have required a fair bit of hacking. Writing custom code was a simpler option, and it resulted in less code (mar_read.c is less than 300 lines of code). Moreover, the update system does not need a standard file format. The elements stored in the archive are bzip2 compressed binary diffs, generated using a variation of bsdiff. So, being able to unpack the archive file using standard tools wouldn't be very useful in and of itself.

MAR extraction tools

There’s 2 different python scripts:

Note:

  • The latter is more recent, but has more dependencies.
  • The former works just fine for extracting files
    • to use the former: cd tmp_dir; mar.py -x ../foo.mar
    • for bzip2'ed files such as b2g mar files: try 'mar.py -x -j fota-flame-update.mar'
    • or 'mv update.zip update.zip.bz2; bunzip2 update.zip.bz2'
Retrieved from "https://wiki.allizom.org/index.php?title=Software_Update:MAR&oldid=1197510"