ReleaseEngineering/Applications/Mapper: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(added links to docs from vcs-sync)
(new mapper url)
 
(17 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= What is it? =
= What is it? =
mapper is a web application that converts hg revisions to a git revision using the hggit map files created as part of the repository conversion.
When we convert hg repositories to git, and vice versa, the hg changeset SHA (the 40 character hexadecimal string that you get when you commit a change) is different to the git commit id (the equivalent SHA used by git).


= Source =
In order to keep track of which hg changeset SHAs relate to which git commit SHAs, we keep a database of the mappings, together with details about the project the SHAs come from, and what time they were inserted into the database.
mapper's source is currently hosted at https://github.com/catlee/mapper


= Deployment =
The vcs sync tool (checked into mozharness) is the tool which performs the conversion between hg repos and git repos, and this is documented separately. It is responsible for performing the conversion - this is outside the scope of mapper.
mapper only requires bottle to run. It's recommended to run inside a virtual environment


= Mapper Data Source =
Mapper is a rest api, that allows:
# insertion of new mappings and projects (a "project" is essentially the name of the repo - e.g. build-tools) (HTTP POST)
# insertion of git/hg mappings for a given project (HTTP POST)
# retrieval of mappings for a given project (HTTP GET)


This section describes (roughly) how vcs-sync provides the map files served by mapper.
Behind the scenes, it is reading/writing from the database (using sqlalchemy).


The map files are generated and combined on the vcs-sync machines, then pulled onto cruncher and used by `mapper`. See the [http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/ source] and [http://people.mozilla.com/~hwine/tmp/vcs2vcs/ docs] for vcs-sync for more details. (Especially the [http://people.mozilla.com/~hwine/tmp/vcs2vcs/mapper_support.html mapper support] section.)
Note: the vcs sync tool is a client of the mapper: it is vcs sync that inserts into mapper (i.e. uses the HTTP POST methods).
The other clients of mapper will be:
# people / developers - wanting to query mappings
# b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests


''Note that the above docs will be integrated into the wiki after vcs-sync development stabilizes.''
Mapper is written as a RelEng API blueprint - please note RelEng API has its [https://docs.mozilla-releng.net/projects/releng-mapper.html own documentation] too.
 
 
= Mapper Development & Deployment =
 
The mapper is a project in [https://github.com/mozilla-releng/services mozilla-releng/services] repository, and so the details about how to deploy locally for development, and where the staging/production environments are, can be seen in the [https://docs.mozilla-releng.net/ RelEngAPI documentation].
 
 
= Mapper Data Feed =
 
VCS Sync is making HTTP Post requests to mapper, providing new mappings for mapper to insert into its database.
 
= API methods =
 
Below is a summary of the 4 unauthenticated API methods currently usable, together with example urls from the production environment.
 
For fuller details, and endpoints requiring authentication, the in process documentation is authoritative: [https://mapper.mozilla-releng.net/docs production] or [https://mapper.staging.mozilla-releng.net/docs staging] instances.
 
== GET Routes ==
 
=== Returns a mapping pair ===
* GET:  /<project>/rev/<vcs_type>/<commit>
* Example: https://mapper.mozilla-releng.net/build-puppet/rev/git/69d64a8a18e6e001eb015646a82bcdaba0e78a24
* Example: https://mapper.mozilla-releng.net/build-puppet/rev/hg/68f1b2b9996c4e33aa57771b3478932c9fb7e161
 
=== Returns full mapfile for a given project ===
* GET:  /<project>/mapfile/full
* Example: https://mapper.mozilla-releng.net/build-puppet/mapfile/full
 
=== Returns a subset of a mapfile, since a given date ===
* GET:  /<project>/mapfile/since/<since>
* Example: https://mapper.mozilla-releng.net/build-mozharness/mapfile/since/29.05.2014%2017:02:09%20CEST
 
=== Returns list of projects ===
* GET:  /projects
* Example: https://mapper.mozilla-releng.net/projects
 
= Staging =
 
The staging environment is available under https://mapper.staging.mozilla-releng.net (Look under "Documentation" to find details.)

Latest revision as of 01:54, 30 June 2018

What is it?

When we convert hg repositories to git, and vice versa, the hg changeset SHA (the 40 character hexadecimal string that you get when you commit a change) is different to the git commit id (the equivalent SHA used by git).

In order to keep track of which hg changeset SHAs relate to which git commit SHAs, we keep a database of the mappings, together with details about the project the SHAs come from, and what time they were inserted into the database.

The vcs sync tool (checked into mozharness) is the tool which performs the conversion between hg repos and git repos, and this is documented separately. It is responsible for performing the conversion - this is outside the scope of mapper.

Mapper is a rest api, that allows:

  1. insertion of new mappings and projects (a "project" is essentially the name of the repo - e.g. build-tools) (HTTP POST)
  2. insertion of git/hg mappings for a given project (HTTP POST)
  3. retrieval of mappings for a given project (HTTP GET)

Behind the scenes, it is reading/writing from the database (using sqlalchemy).

Note: the vcs sync tool is a client of the mapper: it is vcs sync that inserts into mapper (i.e. uses the HTTP POST methods). The other clients of mapper will be:

  1. people / developers - wanting to query mappings
  2. b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests

Mapper is written as a RelEng API blueprint - please note RelEng API has its own documentation too.


Mapper Development & Deployment

The mapper is a project in mozilla-releng/services repository, and so the details about how to deploy locally for development, and where the staging/production environments are, can be seen in the RelEngAPI documentation.


Mapper Data Feed

VCS Sync is making HTTP Post requests to mapper, providing new mappings for mapper to insert into its database.

API methods

Below is a summary of the 4 unauthenticated API methods currently usable, together with example urls from the production environment.

For fuller details, and endpoints requiring authentication, the in process documentation is authoritative: production or staging instances.

GET Routes

Returns a mapping pair

Returns full mapfile for a given project

Returns a subset of a mapfile, since a given date

Returns list of projects

Staging

The staging environment is available under https://mapper.staging.mozilla-releng.net (Look under "Documentation" to find details.)

Retrieved from "https://wiki.allizom.org/index.php?title=ReleaseEngineering/Applications/Mapper&oldid=1196478"