canmove, Confirmed users
1,394
edits
User:Djmitche/New Releng Puppet Infrastructure: Difference between revisions
User:Djmitche/New Releng Puppet Infrastructure (view source)
Revision as of 01:52, 3 September 2011
, 3 September 2011no edit summary
(Created page with "This is a complete re-implementation of puppet for release engineering. = Goals = * A modern puppet installation, completely specifying '''all''' releng infrastructure (includin...") |
No edit summary |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
* Manifests structured to apply settings across all machines, rather than distinct sets of manifests for each slave silo | * Manifests structured to apply settings across all machines, rather than distinct sets of manifests for each slave silo | ||
* Usable by external parties, both inside and outside of mozilla | * Usable by external parties, both inside and outside of mozilla | ||
* Hands-free installations | |||
= General = | = General = | ||
The releng puppet masters are managed by IT (in fact, managed by IT's puppet infrastructure). | The releng puppet masters are managed by IT (in fact, managed by IT's puppet infrastructure). | ||
The releng puppet infrastructure will be using the same puppet versions as the rest of Mozilla. Currently, this is Puppet-2.7.1. As IT upgrades, the masters will be upgraded; releng can then upgrade the clients using puppet itself. | |||
= Base Images = | |||
The base images for this infrastructure are barely-modified OS installs. They have just enough installed that they can connect to a puppet server, get certificates, and puppetize on boot. | |||
See [[User:Djmitche/User:Djmitche/New Releng Puppet Infrastructure/Base Images|the base images page]] for details on how that is set up. | |||
= Manifests = | = Manifests = | ||
The manifests are currently in http://hg.mozilla.org/users/dmitchell_mozilla.com/puppet, but this will move to http://hg.mozilla.org/build/puppet eventually. | The manifests are currently in http://hg.mozilla.org/users/dmitchell_mozilla.com/puppet, but this will move to http://hg.mozilla.org/build/puppet eventually. | ||
= Big Files = | |||
Some files are too big to end up in version control -- mostly package sources. | |||
== Concerns == | |||
* Reproducibility - we should have a source-of-record for all files, and this should be enforced | |||
** {{todo|how?}} | |||
* Security - some of the packages have special licensing, or may have other security concerns; these will need to be protected from unauthorized access | |||
* Stability - releng systems should not dynamically update, even for security updates. We will need to have "snapshots" of constantly-evolving repositories like EPEL or updates, and a way to update those on demand. | |||
* Storage - this can wind up taking a *huge* amount of space. Saving space with hard-links between identical files might help, especially with snapshots. | |||
** {{todo|Is there a utility to fix this?}} | |||
* Replication - how can all of the masters stay in sync? | |||
* Updates - how can we time package updates? | |||
** This has significant security implications | |||
== RPMs / Yum Repos Repos == | |||
There are several repos hosted on each puppet master: | |||
* CentOS 6 | |||
** os - from the DVD | |||
** updates - mirrored at a well-defined time | |||
* EPEL 6 - mirrored at a well-defined time | |||
* releng | |||
** public - packages we can share with the community | |||
** private - packages that are private or proprietary (hopefully minimal) | |||
most of the questions in "Concerns", above, are still un-answered. | |||
== DMGs == | |||
== Python Packages == | |||
== Windows Packages == | |||
= Masters = | = Masters = | ||
| Line 19: | Line 54: | ||
The masters update their manifests from mercurial once every 5 minutes, with a bit of "splay" added (so it does not always occur on the 5-minute mark). Any errors during the update are emailed, as well as a diff of the manifests when they change; the latter forms a kind of change control. | The masters update their manifests from mercurial once every 5 minutes, with a bit of "splay" added (so it does not always occur on the 5-minute mark). Any errors during the update are emailed, as well as a diff of the manifests when they change; the latter forms a kind of change control. | ||
== Multi-Master == | |||
== Cert Signing == | |||
A sysadmin asked the Architect, | |||
"What's the best way to install a new system?" | |||
The Architect answered, | |||
"Turn it on." | |||
The sysadmin was enlightened. | |||
All of our installation tools are scriptable. These tools are responsible for fetching a signed certificate from the puppet master and installing it on the client before its first boot. This transaction will be authenticated using a protected shared secret. Non-Mozilla users can simply omit this part of the setup and sign certificates by hand. | |||
= Clients = | = Clients = | ||
Client images should proceed automatically from imaging to a fully-operational state. The base | Client images should proceed automatically from imaging to a fully-operational state. The base images are designed to support this. | ||
= Automatic/Manual Updates = | |||
A few things will need to be kept current. We will need some automated means to nag about these, lest the releng infra again fall out of date. | |||
* security updates for packages | |||
* puppet versions | |||