MozillaWiki

ReleaseEngineering/Applications/Mapper: Difference between revisions

Page Discussion

ReleaseEngineering/Applications/Mapper (view source)

Revision as of 01:54, 30 June 2018

148 bytes added ,  30 June 2018
new mapper url
(new mapper url)
 
(14 intermediate revisions by 2 users not shown)
Line 7: Line 7:


Mapper is a rest api, that allows:
Mapper is a rest api, that allows:
a) insertion of new mappings and projects (a "project" is essentially the name of the repo - e.g. build-tools) (HTTP POST)
# insertion of new mappings and projects (a "project" is essentially the name of the repo - e.g. build-tools) (HTTP POST)
b) insertion of git/hg mappings for a given project (HTTP POST)
# insertion of git/hg mappings for a given project (HTTP POST)
c) retrieval of mappings for a given project (HTTP GET)
# retrieval of mappings for a given project (HTTP GET)


Behind the scenes, it is reading/writing from the database (using sqlalchemy).
Behind the scenes, it is reading/writing from the database (using sqlalchemy).
Line 15: Line 15:
Note: the vcs sync tool is a client of the mapper: it is vcs sync that inserts into mapper (i.e. uses the HTTP POST methods).
Note: the vcs sync tool is a client of the mapper: it is vcs sync that inserts into mapper (i.e. uses the HTTP POST methods).
The other clients of mapper will be:
The other clients of mapper will be:
a) people / developers - wanting to query mappings
# people / developers - wanting to query mappings
b) b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests
# b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests


Mapper is written as a RelEng API blueprint - please note RelEng API has its own documentation too.
Mapper is written as a RelEng API blueprint - please note RelEng API has its [https://docs.mozilla-releng.net/projects/releng-mapper.html own documentation] too.


= Source =
mapper's source is currently hosted at https://github.com/catlee/mapper


= Deployment =
= Mapper Development & Deployment =
mapper only requires bottle to run. It's recommended to run inside a virtual environment


Our current production deployment of mapper lives on cruncher under /home/buildduty/mapper. It listens locally on port 8888 (specified in mapper/app.py). The apache instance on cruncher is configured to forward requests to http://cruncher/mapper/* to http://locahost:8888/* (configured in /etc/httpd/conf/httpd.conf)
The mapper is a project in [https://github.com/mozilla-releng/services mozilla-releng/services] repository, and so the details about how to deploy locally for development, and where the staging/production environments are, can be seen in the [https://docs.mozilla-releng.net/ RelEngAPI documentation].


The mapper process is managed by supervisord which will ensure it is started up if the machine is ever rebooted, or if the process crashes. (configured in /etc/supervisord.conf)


= Mapper Data Source =
= Mapper Data Feed =


This section describes (roughly) how vcs-sync provides the map files served by mapper.
VCS Sync is making HTTP Post requests to mapper, providing new mappings for mapper to insert into its database.


The map files are generated and combined on the vcs-sync machines, then pulled onto cruncher and used by `mapper`. See the [http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/ source] and [http://people.mozilla.com/~hwine/tmp/vcs2vcs/ docs] for vcs-sync for more details. (Especially the [http://people.mozilla.com/~hwine/tmp/vcs2vcs/mapper_support.html mapper support] section.)
= API methods =


mapper expects hggit map files to be available under the 'mapfiles' directory of the application. On cruncher, these are in /home/buildduty/mapper/mapfiles. Each subdirectory of mapfiles corresponds loosely to a different repository being tracked. On cruncher, the mapfiles for each of these projects are symlinked to the mapfiles being published to cruncher via the process outlined above.
Below is a summary of the 4 unauthenticated API methods currently usable, together with example urls from the production environment.


''Note that the above docs will be integrated into the wiki after vcs-sync development stabilizes.''
For fuller details, and endpoints requiring authentication, the in process documentation is authoritative: [https://mapper.mozilla-releng.net/docs production] or [https://mapper.staging.mozilla-releng.net/docs staging] instances.
 
== GET Routes ==
 
=== Returns a mapping pair ===
* GET:  /<project>/rev/<vcs_type>/<commit>
* Example: https://mapper.mozilla-releng.net/build-puppet/rev/git/69d64a8a18e6e001eb015646a82bcdaba0e78a24
* Example: https://mapper.mozilla-releng.net/build-puppet/rev/hg/68f1b2b9996c4e33aa57771b3478932c9fb7e161
 
=== Returns full mapfile for a given project ===
* GET:  /<project>/mapfile/full
* Example: https://mapper.mozilla-releng.net/build-puppet/mapfile/full
 
=== Returns a subset of a mapfile, since a given date ===
* GET:  /<project>/mapfile/since/<since>
* Example: https://mapper.mozilla-releng.net/build-mozharness/mapfile/since/29.05.2014%2017:02:09%20CEST
 
=== Returns list of projects ===
* GET:  /projects
* Example: https://mapper.mozilla-releng.net/projects
 
= Staging =
 
The staging environment is available under https://mapper.staging.mozilla-releng.net (Look under "Documentation" to find details.)
Garbas
14

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/984436...1196478"