ReleaseEngineering/Applications/Mapper: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
Line 18: Line 18:
# b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests
# b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests


Mapper is written as a RelEng API blueprint - please note RelEng API has its own documentation too.
Mapper is written as a RelEng API blueprint - please note RelEng API has its [https://api.pub.build.mozilla.org/docs/ own documentation] too.


= Source =
= Source =
Line 25: Line 25:
This will be moving as soon as it is ready to go to production. Currently it is in staging (see [https://bugzilla.mozilla.org/show_bug.cgi?id=847640 bug 847640])
This will be moving as soon as it is ready to go to production. Currently it is in staging (see [https://bugzilla.mozilla.org/show_bug.cgi?id=847640 bug 847640])


= Old Mapper =
= Old Mapper Deployment =


Until the "New Mapper" goes live, this is the information about the "old mapper" (which is being superceded).
Until the "New Mapper" goes live, this is the information about the "old mapper" (which is being superceded).
Line 37: Line 37:
Source is available at https://github.com/catlee/mapper
Source is available at https://github.com/catlee/mapper


= Mapper Data Source =
= New Mapper Deployment =
 
The new mapper is a Releng API blueprint, and so the details about how to deploy locally for development, and where the staging/production environments are, can be seen in the [https://api.pub.build.mozilla.org/docs/ RelEngAPI documentation].
 
= Old Mapper Data Feed =


This section describes (roughly) how vcs-sync provides the map files served by mapper.
This section describes (roughly) how vcs-sync provides the map files served by mapper.
Line 46: Line 50:


''Note that the above docs will be integrated into the wiki after vcs-sync development stabilizes.''
''Note that the above docs will be integrated into the wiki after vcs-sync development stabilizes.''
= New Mapper Data Feed =
VCS Sync is making HTTP Post requests to mapper, providing new mappings for mapper to insert into its database.
= API methods =
Below are the 7 API methods currently supported, together with example urls from the staging environment.
Please note, mapper uses secure authentication (provided by RelEng API), so you will not be able to use the POST examples below, but you will be able query mapper using e.g. the example GET requests below.
== GET Routes ==
=== Returns a mapping pair ===
* GET:  /<project>/rev/<vcs_type>/<commit>
* Example: https://api-pub-build.allizom.org/mapper/build-puppet/rev/git/69d64a8a18e6e001eb015646a82bcdaba0e78a24
* Example: https://api-pub-build.allizom.org/mapper/build-puppet/rev/hg/68f1b2b9996c4e33aa57771b3478932c9fb7e161
=== Returns full mapfile for a given project ===
* GET:  /<project>/mapfile/full
* Example: https://api-pub-build.allizom.org/mapper/build-puppet/mapfile/full
=== Returns a subset of a mapfile, since a given date ===
* GET:  /<project>/mapfile/since/<since>
* Example: https://api-pub-build.allizom.org/mapper/build-mozharness/mapfile/since/29.05.2014%2017:02:09%20CEST
== POST Routes ==
=== Inserts a mappings file strictly (no duplicates allowed) ===
* POST: /<project>/insert
* Example: https://api-pub-build.allizom.org/mapper/insert
=== Inserts a mappings file, allowing duplicates ===
* POST: /<project>/insert/ignoredups
* Example: https://api-pub-build.allizom.org/mapper/insert/ignoredups
=== Inserts an individual mapping ===
* POST:  /<project>/insert/<git_commit>/<hg_changeset>
* Example: https://api-pub-build.allizom.org/mapper/insert/69d64a8a18e6e001eb015646a82bcdaba0e78a24/68f1b2b9996c4e33aa57771b3478932c9fb7e161
=== Inserts a new project ===
* POST:  /<project>
* Example: https://api-pub-build.allizom.org/mapper/build-puppet

Revision as of 19:14, 30 May 2014

What is it?

When we convert hg repositories to git, and vice versa, the hg changeset SHA (the 40 character hexadecimal string that you get when you commit a change) is different to the git commit id (the equivalent SHA used by git).

In order to keep track of which hg changeset SHAs relate to which git commit SHAs, we keep a database of the mappings, together with details about the project the SHAs come from, and what time they were inserted into the database.

The vcs sync tool (checked into mozharness) is the tool which performs the conversion between hg repos and git repos, and this is documented separately. It is responsible for performing the conversion - this is outside the scope of mapper.

Mapper is a rest api, that allows:

  1. insertion of new mappings and projects (a "project" is essentially the name of the repo - e.g. build-tools) (HTTP POST)
  2. insertion of git/hg mappings for a given project (HTTP POST)
  3. retrieval of mappings for a given project (HTTP GET)

Behind the scenes, it is reading/writing from the database (using sqlalchemy).

Note: the vcs sync tool is a client of the mapper: it is vcs sync that inserts into mapper (i.e. uses the HTTP POST methods). The other clients of mapper will be:

  1. people / developers - wanting to query mappings
  2. b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests

Mapper is written as a RelEng API blueprint - please note RelEng API has its own documentation too.

Source

mapper's source is currently hosted at https://github.com/petemoore/mapper

This will be moving as soon as it is ready to go to production. Currently it is in staging (see bug 847640)

Old Mapper Deployment

Until the "New Mapper" goes live, this is the information about the "old mapper" (which is being superceded).

mapper only requires bottle to run. It's recommended to run inside a virtual environment

Our current production deployment of mapper lives on cruncher under /home/buildduty/mapper. It listens locally on port 8888 (specified in mapper/app.py). The apache instance on cruncher is configured to forward requests to http://cruncher/mapper/* to http://locahost:8888/* (configured in /etc/httpd/conf/httpd.conf)

The mapper process is managed by supervisord which will ensure it is started up if the machine is ever rebooted, or if the process crashes. (configured in /etc/supervisord.conf)

Source is available at https://github.com/catlee/mapper

New Mapper Deployment

The new mapper is a Releng API blueprint, and so the details about how to deploy locally for development, and where the staging/production environments are, can be seen in the RelEngAPI documentation.

Old Mapper Data Feed

This section describes (roughly) how vcs-sync provides the map files served by mapper.

The map files are generated and combined on the vcs-sync machines, then pulled onto cruncher and used by `mapper`. See the source and docs for vcs-sync for more details. (Especially the mapper support section.)

mapper expects hggit map files to be available under the 'mapfiles' directory of the application. On cruncher, these are in /home/buildduty/mapper/mapfiles. Each subdirectory of mapfiles corresponds loosely to a different repository being tracked. On cruncher, the mapfiles for each of these projects are symlinked to the mapfiles being published to cruncher via the process outlined above.

Note that the above docs will be integrated into the wiki after vcs-sync development stabilizes.

New Mapper Data Feed

VCS Sync is making HTTP Post requests to mapper, providing new mappings for mapper to insert into its database.

API methods

Below are the 7 API methods currently supported, together with example urls from the staging environment.

Please note, mapper uses secure authentication (provided by RelEng API), so you will not be able to use the POST examples below, but you will be able query mapper using e.g. the example GET requests below.

GET Routes

Returns a mapping pair

Returns full mapfile for a given project

Returns a subset of a mapfile, since a given date

POST Routes

Inserts a mappings file strictly (no duplicates allowed)

Inserts a mappings file, allowing duplicates

Inserts an individual mapping

Inserts a new project

Retrieved from "https://wiki.allizom.org/index.php?title=ReleaseEngineering/Applications/Mapper&oldid=984451"