ReleaseEngineering/Applications/Slavealloc: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
(deleting obsolete page)
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Application Description ==
Slavealloc is a client-server application.  The client is [http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildslave/files/runslave.py runslave.py].  Communication is via a very basic HTTP request to http://slavealloc.build.mozilla.org/production/gettac/$slavename, where the response is expected to be a <tt>buildbot.tac</tt> file suitable for use to start buildslave.


The slavealloc server is a implemented as a small Twisted application ([http://hg.mozilla.org/build/tools/file/tip/lib/python/slavealloc source]) which serves the tac generator, a JSON REST interface, and a client-side JavaScript interface.
The same source code also implements a command-line frontend to the REST interface.
== Requirements ==
The server depends on
* MySQL database
* Python
** Twisted
** SQLAlchemy
== External Resources ==
The slavealloc server uses the following external resources:
* two MySQL databases (production and staging)
* VM host for the slaveallocator VM
== Security ==
The slave allocator hands out low-security slave passwords in the .tac files, which are stored in cleartext in the database.  It does not do any sort of authentication either for read or modify operations, and relies on the Build VLAN firewalls to prevent external access.
== Development ==
Install tools in a virtualenv:
cd tools
virtualenv sandbox
sandbox/bin/pip install -e .
Then you can run the slavealloc daemon locally from the root of the tools repository with a simple:
sandbox/bin/twistd -noy lib/python/slavealloc/contrib/slavealloc-combined.tac
Note that due to what I believe to be a bug in pip, you may need to explicitly install Twisted to get the twistd executable installed:
pip install -U twisted
To set up a fresh database, use
sandbox/bin/slavealloc dbinit -D sqlite:///slavealloc.db
This configuration will use SQLite to access {{{slavealloc.db}}} in the current directory.  You can hack on the static web content while the daemon is running.
== Monitoring ==
none yet ({{bug|633365}})
== Deployment ==
The slave allocator server is deployed on a single host, <tt>slavealloc.build.mozilla.org</tt>.
=== Server Setup ===
IT installed RHEL6 along with MySQL client libraries, and set up the proper firewall rules to allow database access.
As root, virtualenv-1.5.2 was installed into the system Python library.  The following system packages were installed via yum:
* http://download.fedora.redhat.com/pub/epel/6/i386/epel-release-6-5.noarch.rpm (for EPEL packages; use --nogpgcheck)
* hg
* MySQL-python
* nginx (from EPEL repo)
iptables was configured to only allow incoming connections on TCP ports 80 and 22, along with ICMP.
=== Nginx ===
Nginx frontends for both staging and production instances.  The virtualhosts files are available [http://hg.mozilla.org/build/tools/file/828895e6a5d3/lib/python/slavealloc/contrib in hg].
Note that on the x86_64 system slavealloc is currently installed on, the following must be added to the <tt>http</tt> section of <tt>nginx.conf</tt>:
    # required to use virtualhosts - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=547722
    server_names_hash_bucket_size 33;
=== Twisted Daemon ===
The 'slavealloc' user runs the twisted daemon on this host.  The user account is locked and accessed only via su from root.
The daemon is installed in a virtualenv at <tt>/tools/slavealloc-$rev</tt>, using the pre-checked python packages on the puppet server.  Note that <tt>--no-site-packages</tt> is ''not'' used here, because we need access to the (binary) MySQL-python package which is installed systemwide:
cd /tools
virtualenv slavealloc-8fe4dbc09d03
ln -s slavealloc-8fe4dbc09d03 slavealloc
./slavealloc/bin/pip install -e hg+http://hg.mozilla.org/build/tools@8fe4dbc09d03#egg=tools \
    --no-index --find-links=http://staging-puppet.build.mozilla.org/staging/python-packages/
Once this was set up, the 'slavealloc dbinit' command was used to initialize the database.
The production and staging tac files are in <tt>/build/slavealloc</tt>.  Staging runs on port 1079, and production runs on 1080.  The files are similar to those [http://hg.mozilla.org/build/tools/file/828895e6a5d3/lib/python/slavealloc/contrib in hg].
Startup is done via initscripts, the initscripts are [http://hg.mozilla.org/build/tools/file/828895e6a5d3/lib/python/slavealloc/contrib/ in hg].
=== Slave Side ===
All slaves run [http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildslave/files/runslave.py runslave.py] during startup.  This file is distributed via puppet. The larger slave-startup process is described in [[ReleaseEngineering/Buildslave Startup Process]].
== See Also ==
See [[User:Djmitche/Slave Allocator Proposal]]

Latest revision as of 21:49, 19 November 2018

Retrieved from "https://wiki.allizom.org/index.php?title=ReleaseEngineering/Applications/Slavealloc&oldid=1204135"