MozillaWiki

ReferencePlatforms/Linux-CentOS-5.0: Difference between revisions

Page Discussion

ReferencePlatforms/Linux-CentOS-5.0 (view source)

Revision as of 14:32, 10 April 2012

11,850 bytes added ,  10 April 2012
→‎Install EPEL
(Moved some post-install instructions into the 'already done for you' section to reflect changes to reference image)
 
(95 intermediate revisions by 11 users not shown)
Line 1: Line 1:
This procedure will re-create the CentOS 5.0-based reference VM image. The current version is '''Version 5'''. Internally, this VM is named '''CentOS-5.0-ref-tools-vm'''.
This procedure will re-create the CentOS 5.0-based reference VM image. Internally, this VM is named '''CentOS-5.0-ref-tools-vm'''.


= Things Done For You =
= Things Done For You =
Line 11: Line 11:
  0c3990be2271bf44c1495aa0003b5552  CentOS-5.0-i386-bin-5of6.iso
  0c3990be2271bf44c1495aa0003b5552  CentOS-5.0-i386-bin-5of6.iso
  9e6f91a5292f46b02777133765fc03fe  CentOS-5.0-i386-bin-6of6.iso
  9e6f91a5292f46b02777133765fc03fe  CentOS-5.0-i386-bin-6of6.iso
TB: Thunderbird build clients use a separate /buildbot partition of 80GB.  Now that we reboot after each build, be sure to "tune2fs -c 0" the buildbot partition to avoid an fsck of it every 20 reboots.


=== Package Selection ===
=== Package Selection ===
Line 24: Line 26:
* '''Applications'''
* '''Applications'''
** Editors
** Editors
** Test-based internet
** Text-based internet
*** Add lynx
*** Add lynx
* '''Development'''
* '''Development'''
Line 104: Line 106:
  make install 2>&1 | tee /tools/dist/logs/python2.5.1-make_install.log
  make install 2>&1 | tee /tools/dist/logs/python2.5.1-make_install.log
  cd /tools/
  cd /tools/
  ln -s python2.5.1 python
  ln -s python-2.5.1 python


== Version 3 ==
== Version 3 ==
Line 181: Line 183:
  cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -d nagios mozilla/tools/nagios
  cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -d nagios mozilla/tools/nagios
  mv nagios/nrpe-linux.cfg /etc/nagios/nrpe.cfg
  mv nagios/nrpe-linux.cfg /etc/nagios/nrpe.cfg
=== ''Installing Autoconf 2.13'' ===
{{bug|433492}} added autoconf 2.13 on 30/June/2008, which puts it somewhere between version 4 and 5.
sudo yum install autoconf213


== Version 5 ==
== Version 5 ==
Line 272: Line 279:
  ln -s gcc-4.1.1 gcc
  ln -s gcc-4.1.1 gcc


== Version 6 ==


= Post-Install Setup =
=== Move scratchbox ===
Post-install instructions are written for the latest version (currently, Version 5).<br />
You should have a /builds drive added to the VM before completing this section.
You should have a /builds drive added to the VM before completing this section.
All of this should be run as root.
All of this should be run as root.
Line 300: Line 307:
</pre></code>
</pre></code>


== Installing BuildBot ==
== Version 7 ==
 
20090112 16:30 PST
 
=== metacity / Xvfb crontabs ===
 
Add the following to cltbld's crontab:
<pre>
# Make sure Xvfb is running on :2
@reboot    ps -C Xvfb | grep -q Xvfb || exec Xvfb :2 -screen 0 1280x1024x24 &
*/5 * * * * ps -C Xvfb | grep -q Xvfb || exec Xvfb :2 -screen 0 1280x1024x24 &
 
# Make sure metacity is running on :2
@reboot    ps -C metacity -f | grep -q :2 || exec metacity --display :2 --replace &
*/5 * * * * ps -C metacity -f | grep -q :2 || exec metacity --display :2 --replace &
</pre>
 
== Version 8 ==
 
20090122 9:30 PST
 
=== upgraded mercurial to 1.1.2 ===
Updating to newer mercurial version using this script:
<pre>
#!/bin/sh
 
pushd /tools
wget http://www.selenic.com/mercurial/release/mercurial-1.1.2.tar.gz
tar -zxvf mercurial-1.1.2.tar.gz
cd mercurial-1.1.2
export PYTHONHOME="/tools/python"
export PATH="/tools/python/bin:$PATH"
python setup.py install
rm -rf /tools/mercurial-1.1.2*
 
echo "Hg update complete"
 
</pre>
 
== Version 9 ==
 
20090128 18:15 PST
 
=== install libnotify-devel rpm ===
 
If yum-utils are not installed yet, do the following:
 
wget http://vault.centos.org/5.1/os/i386/CentOS/yum-utils-1.0.4-3.el5.centos.2.noarch.rpm
rpm -ivh yum-utils-1.0.4-3.el5.centos.2.noarch.rpm
 
Developer requested devel package for libnotify {{bug|473831}}
# root
yumdownloader libnotify-devel-0.4.2-6.el5
rpm -ivh libnotify-devel-0.4.2-6.el5.i386.rpm
yum list installed | grep libnotify
  libnotify.i386                0.4.2-6.el5            installed     
  libnotify-devel.i386          0.4.2-6.el5            installed
 
== Version 10 ==
=== Buildbot init scripts ===
cd /builds
mkdir slave
su -
cd /etc/default
wget -Obuildbot https://bugzilla.mozilla.org/attachment.cgi?id=359078
cd /etc/init.d
wget -Obuildbot https://bugzilla.mozilla.org/attachment.cgi?id=359080
chmod +x buildbot
chkconfig --add buildbot
/etc/init.d/buildbot start
sudo -e /etc/sudoers
# Add 'cltbld ALL=NOPASSWD: /usr/bin/reboot' as the last line, save and exit
 
=== Update scratchbox mercurial and python ===
 
First, fix the DNS in scratchbox.
 
cp /etc/resolv.conf /scratchbox/users/cltbld/targets/CHINOOK_X86/etc/resolv.conf
cp /etc/resolv.conf /scratchbox/users/cltbld/targets/CHINOOK_ARMEL/etc/resolv.conf
cp /etc/resolv.conf /scratchbox/users/cltbld/targets/CHINOOK-ARMEL-2007/etc/resolv.conf
 
Allow cltbld to write to the tool areas
 
su -
chown -R cltbld /scratchbox/devkits/maemo3-tools
exit
 
Initial setup
 
/scratchbox/moz_scratchbox
wget http://people.mozilla.org/~asasaki/mercurial-1.1.2.tar.gz
tar zxvf mercurial-1.1.2.tar.gz
echo 'export PYTHONPATH=/usr/lib/python2.5/site-packages:$PYTHONPATH' >> .bashrc
 
Do the following for CHINOOK_X86, while still running <tt>/scratchbox/moz_scratchbox</tt>:
 
sb-conf select CHINOOK_X86
echo 'deb http://repository.maemo.org/extras/ bora free' >> /etc/apt/sources.list
apt-get update
yes | apt-get install python2.5 python2.5-dev python2.5-xml
cd /usr/bin
rm python; ln -s python2.5 python
cd ~/mercurial-1.1.2
python2.5 setup.py install
cd /scratchbox/devkits/maemo3-tools/bin
mv hg hg.old
mv hgmerge hgmerge.old
ln -s /usr/bin/hg .
hg --version
 
The <tt>hg --version</tt> should give you no errors.
 
Repeat for CHINOOK_ARMEL... but I was unsuccessful getting hg 1.1.2 installed/working there.
 
sb-conf select CHINOOK_ARMEL
echo 'deb http://repository.maemo.org/extras/ bora free' >> /etc/apt/sources.list
apt-get update
yes | apt-get install python2.5 python2.5-dev python2.5-xml
cd /usr/bin
rm python; ln -s python2.5 python
 
Repeat for CHINOOK-ARMEL-2007.
 
sb-conf select CHINOOK-ARMEL-2007
echo 'deb http://repository.maemo.org/extras/ bora free' >> /etc/apt/sources.list
apt-get update
yes | apt-get install python2.5 python2.5-dev python2.5-xml
cd /usr/bin
rm python; ln -s python2.5 python
cd ~/mercurial-1.1.2
python2.5 setup.py install
cd
hg --version
rm -rf mercurial-1.1.2*
 
== Version 11 ==
=== ESD Wrapper script ===
Because of {{bug|469635}} we need to wrap the esd binary to toss out stderr.
cd /usr/bin
mv esd esd.real
wget -Oesd https://bug469635.bugzilla.mozilla.org/attachment.cgi?id=363722
chmod +x esd
 
== Version 12 ==
=== Install libxdamage-dev and libxcomposite-dev in Scratchbox ===
/scratchbox/moz_scratchbox
sb-conf select CHINOOK_X86
apt-get install --force-yes -y libxdamage-dev libxcomposite-dev
sb-conf select CHINOOK_ARMEL
apt-get install --force-yes -y libxdamage-dev libxcomposite-dev
sb-conf select CHINOOK-ARMEL-2007
apt-get install --force-yes -y libxdamage-dev libxcomposite-dev
 
== Version 13 ==
=== Install libhildon1-dev to scratchbox ===
/scratchbox/moz_scratchbox
apt-get install --force-yes -y libhildon1-dev
 
== Version 14 ==
=== Disable Xvnc and some cronjobs ===
The updatedb cronjob manages the 'locate' database, which we do not use, and causes increased load every night at 4am PDT. The 'prelink' cronjob prelinks binaries for faster loading. This probably doesn't help build times in any significant way because we call the same binaries over and over during the build process. The Xvnc session is totally unused. To disable them, do the following:
# as root
chkconfig vncserver off
cd /etc/cron.daily
mv mlocate.cron ~/
mv prelink ~/
 
== Version 15 ==
=== Install code coverage tools ===
Installed lcov from http://downloads.sourceforge.net/ltp/lcov-1.7-1.noarch.rpm
 
# as root
wget http://downloads.sourceforge.net/ltp/lcov-1.7-1.noarch.rpm
rpm -i lcov-1.7-1.noarch.rpm


Because our Buildbot code changes fairly regularly we don't keep a copy of it on the VM.
Copied genhtml and geninfo from /usr/bin into /usr/local/bin, and replaced all instances of 'die' with 'print'
  export PYTHONPATH=".:/tools/zope-interface/lib/python2.5/site-packages/:/tools/twisted-core/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages/:$PYTHONPATH"
 
  export PYTHONHOME="/tools/python"
# as root
  export PATH="/tools/python/bin:/tools/twisted/bin:/tools/twisted-core/bin:$PATH"
  cp /usr/bin/genhtml /usr/bin/geninfo /usr/local/bin
sed -i 's,die,print,g' /usr/local/bin/genhtml
sed -i 's,die,print,g' /usr/local/bin/geninfo
 
Install jscoverage into /usr/local/bin
 
# as root
wget http://siliconforks.com/jscoverage/download/jscoverage-0.4.tar.bz2
tar jxf jscoverage-0.4.tar.bz2
cd jscoverage-0.4
./configure
make
cp jscoverage jscoverage-server /usr/local/bin
 
== Version 16 ==
=== Install wireless tools ===
 
As root,
yum -y install wireless-tools-devel
 
== Version 17 ==
=== Add noatime option ===
Edit /etc/fstab so that the mount options for / and /builds are 'defaults,noatime'.
 
You don't have to reboot for these changes to take effect, instead you can do,
mount / -o remount,noatime
mount /builds -o remount,noatime
 
== Version 18 ==
=== Install libnotify for scratchbox ===
 
/scratchbox/moz_scratchbox
  apt-get install --force-yes -y libnotify-dev
 
== Version 19 ==
=== Fix scratchbox wireless devel ===
 
/scratchbox/moz_scratchbox -p 'apt-get install --force-yes -y libiw-dev'
 
 
== Version 20 ==
=== Install Puppet ===
==== Install EPEL ====
As root,
  rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
 
==== Move the JDK ====
To make the Puppet manifests a bit easier to manage we need to move the JDK to be in the same format as the other tools dirs (/tools/$name-$version):
# as root
  cd /tools
  cd /tools
  cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -d buildbot-077 mozilla/tools/buildbot
  mv jdk1.5.0_10 jdk-1.5.0_10
cd /tools/buildbot-077
rm -f jdk
  python setup.py install --prefix=/tools/buildbot-077
ln -s jdk-1.5.0_10 jdk
  cd /tools; ln -s buildbot-077 buildbot
 
  # buildbotcustom is only required if you're running a Buildbot master
==== Mount the file share ====
  mkdir buildbotcustom
Mount the puppet-files directory by adding an entry to fstab:
  cd buildbotcustom
# IP is 10.2.71.136 for regular build slaves and 10.2.76.136 for try
  cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -d buildbotcustom mozilla/tools/buildbotcustom
$IP:/export/buildlogs/puppet-files /N  nfs ro 0 0
And then run,
mkdir /N
mount -a
to mount it.
 
==== Install Puppet ====
yum install ruby facter puppet ruby-shadow augeas-libs ruby-augeas
 
Now, we need to run puppet once and sign its cert on the puppetmaster
# On the client
# $hostname == staging-puppet or production-puppet
puppetd --test --server $hostname.build.mozilla.org
You should get a message like "notice: Set to run 'one time'; exiting with no certificate". Sign it on the puppetmaster as follows:
# root @ staging-puppet or production-puppet
puppetca --sign $hostname.build.mozilla.org
 
Setup the sysconfig file:
* Open /etc/sysconfig/puppet in an editor
* replace 'puppet' with production-puppet.build.mozilla.org
 
== Post-puppet packages ==
Since the deployment of Puppet, future changes to the ref platform are management by it. Because of this, providing _exact_ installation instructions can be tricky. All of the packages listed below are installed on the ref platform and some installation guidelines are provided. Looking at [http://hg.mozilla.org/build/puppet-manifests the puppet-manifests repository] may help if you are having trouble.
=== jscoverage ===
* Build from revision 453 of [http://svn.siliconforks.com/jscoverage/trunk/ the jscoverage SVN repository]
* 'jscoverage' and 'jscoverage-server' binaries installed to /usr/local/bin
 
=== GCC 4.3.3 ===
GCC 4.3.3 was installed alongside the other GCC for use on newer branches:
* Get http://ftp.gnu.org/gnu/gcc/gcc-4.3.3/gcc-4.3.3.tar.bz2 and unpack it to /tools/gcc-4.3.3/source
* Pull the patch series from http://hg.mozilla.org/users/tglek_mozilla.com/gcc-moz-plugin-mq/
* Apply the following patches to the GCC sources using git-apply (patch -p1 will probably work as well):
** gty.diff
** plugin.diff
** parm_attrs.diff
** c_process_decl.diff
* mkdir /tools/gcc-4.3.3/build
* cd /tools/gcc-4.3.3/build
* /tools/gcc-4.3.3/source/configure --prefix=/tools/gcc-4.3.3/installed --enable-__cxa_atexit --enable-languages=c,c++
* make install
* The static analysis will require all of /tools/gcc-4.3.3 including the source/build/installed subdirectories
 
=== libhildonfm2-dev ===
Installed inside of scratchbox with the following command:
su - cltbld -c '/scratchbox/moz_scratchbox -p apt-get --yes --force-yes install libhildonfm2-dev
 
=== libconic0-dev ===
Installed inside of scratchbox with the following command:
su - cltbld -c '/scratchbox/moz_scratchbox -p apt-get --yes --force-yes install libconic0-dev'
 
=== access keys from within scratchbox ===
/bin/mount --bind /scratchbox/users/cltbld/home/cltbld/.ssh /home/cltbld/.ssh/
 
=== Upgrade Scratchbox ===
Roughly, the Scratchbox upgrade was done with the following set of commands:
su -
unset PYTHONHOME
unset PYTHONPATH
export PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
umount /home/cltbld/.ssh #until no longer mounted
umount /builds/slave #might not be mounted
/scratchbox/sbin/sbox stop
/scratchbox/sbin/sbox_umount_all
sh maemo-scratchbox-install_5.0.sh -U -s /scratchbox
/etc/init.d/sbox start
exit
sh maemo-sdk-install_5.0.sh -y -d -s /scratchbox
/scratchbox/login
echo deb http://repository.maemo.org/fremantle/<token> nokia-binaries >> /etc/apt/sources.list
fakeroot apt-get install autoconf2.13 libnotify-dev libIDL-dev libqt4-dev
exit
/scratchbox/moz_scratchbox
sb-conf select FREMANTLE_ARMEL
  wget http://www.python.org/ftp/python/2.5.5/Python-2.5.5.tar.bz2
tar jxf Python-2.5.5.tar.bz2
cd Python-2.5.5
CC='host-gcc' --prefix=/usr/local
make -j4
/scratchbox/moz_scratchbox
openssl md5 openssl-0.9.8m.tar.gz Python-2.5.5.tar.bz2
#MD5(openssl-0.9.8m.tar.gz)= 898bf125370926d5f692a2201124f8ec
#MD5(Python-2.5.5.tar.bz2)= 1d00e2fb19418e486c30b850df625aa3
tar jxf Python-2.5.5.tar.bz2
tar zxf openssl-0.9.8m.tar.gz
  cd openssl-0.9.8m
CC=host-gcc CXX=host-g++ ./config shared --prefix=/usr/local
  make -j4
  make install
  cd Python-2.5.5
  CC=host-gcc CXX=host-g++ ./configure --prefix=/host_usr
make -j4
make altinstall
find /usr/local -name "*.so*" -exec ldd {} \; | grep "not found" # should have no output
 
= Post-Install Setup =
Post-install instructions are written for the latest version.<br />
 
== Add slave(s) to configuration files ==
The following files need to be updated with new slaves (note that the production Buildbot config is explicitly not updated here, it will be done later):
* [http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2-staging/config.py buildbot-configs/mozilla2-staging/config.py]
* [http://hg.mozilla.org/build/puppet-manifests/file/tip/site-production.pp puppet-manifests/site-production.pp] (or site-staging.pp for slaves that will be in staging permanently).


When you create the slave, be sure to set "umask = 002" in the buildbot.tac and restart it (otherwise we'll start pushing update snippets around with mode 600).
* For a permanent staging slave - edit /etc/sysconfig/puppet to staging-puppet


== To build mozilla ==
Once your patches have landed, you need to update the following checkouts:
* On staging-master:
# as cltbld
cd /builds/buildbot/user-configs/clean-configs
hg pull && hg up
cd ../../user-configs2/clean-configs
hg pull && hg up
* On production-puppet (or staging-puppet if you updated site-staging.pp):
# as root
cd /etc/puppet/manifests
hg pull && hg up
* If a machine has been recloned/replaced then you also do this as root@production-puppet
puppetca --clean moz2-linux-slaveNN.build.mozilla.org


The mozconfig for building mozilla should contain the following settings:
== Buildbot Setup ==
The buildbot.tac file will automatically be generated to send the slave to the correct staging master (staging-master or sm-staging-try-master).


CC=/tools/gcc/bin/gcc
===Run the slave through staging===
CXX=/tools/gcc/bin/g++
see [[ReleaseEngineering:BuildSlaveSetup]]
Rail
Confirmed users
3,104

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/115622...418390"