QA/Infrastructure/Automation Servers

From MozillaWiki
Jump to navigation Jump to search

The automation infrastructure is the set of machines available for Mozmill-based update and release testing for Firefox.

This is divided into the following sets of machines which all live in the QA vlan in Mountain View (accessible as .qa.mtv1.mozilla.com).

The Project Overview has moved to QA/Infrastructure/Automation_Servers/Project_Overview.

Mac Pros

Mac Pros.png

There are three existing Mac Pros that are legacy hardware for automation runs:

  • QA-Horus aka release1.qa.mtv1.mozilla.com
  • QA-Set aka release2.qa.mtv1.mozilla.com
  • QA-Masterblaster aka release3.qa.mtv1.mozilla.com

All three machines run the latest version of OS X 10.6.

They are located at: 

 Room 2.IDF in Mountain View

All machines have an administrator account with the username, 'mozillaqa'. Access can be done over SSH or Apple Remote Desktop/VNC.

Mac Minis

To cover tests on OS X, 10 Mac Minis were new purchased for this project.

Their specifications are: 

 2.7GHz Dual-Core Intel Core i7
 8GB 1333MHz DDR3 SDRAM - 2x4GB
 500GB Serial ATA Drive @ 5400 rpm

They are located at: 

 Room 2.IDF in Mountain View

In addition, four OS X 10.6 Mac Minis were acquired from ex-RelEng stock.

QA-Mozmill is an older, legacy, Mac Mini previously used for OS X 10.7 testing and as a sandbox.

All machines except QA-Mozmill have an administrator account with the username, "QA". Access can be done over SSH or Apple Remote Desktop/VNC.

The detailed list of machines is as follows:

  • QA-Mozmill - a legacy machine used for 10.7 testing, account is "mozillqa" for this machine.
  • OS X 10.5
    • release1-osx-105.qa.mtv1.mozilla.com (Mozmill CI)
    • release2-osx-105.qa.mtv1.mozilla.com (Mozmill CI)
    • release3-osx-105.qa.mtv1.mozilla.com (Mozmill CI)
  • OS X 10.6
    • release1-osx-106.qa.mtv1.mozilla.com (Mozmill CI)
    • release2-osx-106.qa.mtv1.mozilla.com (Mozmill CI)
    • release3-osx-106.qa.mtv1.mozilla.com (Mozmill CI)
    • release4-osx-106.qa.mtv1.mozilla.com (Mozmill CI - currently used for l10n tests)
  • OS X 10.7
    • release1-osx-107.qa.mtv1.mozilla.com (Mozmill CI)
    • release2-osx-107.qa.mtv1.mozilla.com (Mozmill CI)
    • release3-osx-107.qa.mtv1.mozilla.com (Mozmill CI)
    • release4-osx-107.qa.mtv1.mozilla.com (Selenium - doesn't respond. See bug 774609)
  • OS X 10.8
    • release5-osx-107.qa.mtv1.mozilla.com (Mozmill CI)
    • release6-osx-107.qa.mtv1.mozilla.com (Mozmill CI)
    • release7-osx-107.qa.mtv1.mozilla.com (Mozmill CI - currently used in staging environment)
    • release8-osx-107.qa.mtv1.mozilla.com (Selenium)

QA MV ESX Cluster (HP 1230 Racked Servers)

These servers run ESXi and have a series of Windows and Linux virtual machines running within each.

The hardware specifications are: 

HP DL120 G7 1U Form factor
Intel® Xeon® processor E3-Intel® Xeon® E3-1230 (3.20GHz/4-core/8MB/80W)
1333, HT, Turbo 1/2/3/4 Processor
1 X 250GB SATA HD
16 GB RAM

They are located at: 

 Room 3.IDF in Mountain View

Numbering here starts with '4' since 1 through 3 are used for the existing Mac Pros running VMware Fusion.

Usage

  • release4.qa.mtv1.mozilla.com
    • mm-ub-1204-32-1 (10.250.73.249)
    • mm-ub-1204-32-2 (10.250.73.X)
    • mm-ub-1204-32-3 (10.250.73.X)
    • mm-ub-1204-64-1 (10.250.73.247)
    • mm-ub-1204-64-2 (10.250.73.X)
    • mm-ub-1204-64-3 (10.250.73.X)
  • release5.qa.mtv1.mozilla.com
    • mm-win-2000-1 (10.250.73.252)
    • mm-win-2000-2 (10.250.73.X)
    • mm-win-2000-3 (10.250.73.X)
    • mm-win-xp-1 (10.250.73.254)
    • mm-win-xp-2 (10.250.73.X)
    • mm-win-xp-3 (10.250.73.X)
    • mm-win-vista-1 (10.250.73.251)
    • mm-win-vista-2 (10.250.73.X)
    • mm-win-vista-3 (10.250.73.X)
  • release6.qa.mtv1.mozilla.com
    • mm-win-7-32-1 (10.250.73.244)
    • mm-win-7-32-2 (10.250.73.X)
    • mm-win-7-32-2 (10.250.73.X)
    • mm-win-7-64-1 (10.250.73.253)
    • mm-win-7-64-2 (10.250.73.X)
    • mm-win-7-64-3 (10.250.73.X)
  • release7.qa.mtv1.mozilla.com
  • release8.qa.mtv1.mozilla.com
  • release9.qa.mtv1.mozilla.com
  • release10.qa.mtv1.mozilla.com
    • wq-win-7-32-1 (10.250.73.248)
  • release11.qa.mtv1.mozilla.com
    • esx-win-xp-adm (10.250.73.241)
    • mm-ub-ci-master (10.250.73.243)

These are controlled as a unified vCenter ESX cluster. This is accessible via vSphere as vc1.qa.mtv1.mozilla.com. Download the appropriate vSphere client from: https://vc1.qa.mtv1.mozilla.com/client/VMware-viclient.exe. You can try the web client at: https://vc1.qa.mtv1.mozilla.com:9443/vsphere-client/. You will need an account created on vc1 if you wish to be an administrator on it. This is controlled by Dan Parsons (:lerxst) and should be done through an IT request bug.

Handling of the ESX cluster

To ensure that new and existent VMs and templates are handled correctly you can find some details below. This part only affects administrators of the ESX cluster. Others should file a Mozilla QA / Infrastructure bug to get issues solved.

Templates

All templates for VMs have been created to be run on release11. Their data storage has been set to fs2-qa1 (which is smb://fs2.private.mtv1.mozilla.com/qa1). That means no data is stored on the VM itself but on the shared 1TB network drive instead. Those data should remain on this disk and this storage has to be used for newly created templates. That allows us to share the templates across the different machines in the ESX cluster.

Creating a new template
  • Log into vsphere via the local client
  • Select "Inventory -> VMs and Templates"
  • Right click the templates folder under "qa.mtv1" and select "New Virtual Machine"
  • Select the Custom mode
  • Enter the name of the template which correlates to the platform you want to install, e.g. ubuntu_12.04_32bit
  • As execution host select release11
  • The storage has to be set to fs2-qa1
  • Click through by using the defaults
  • Select the os you want to install
  • Click through by using the defaults
  • For memory use 1GB for 32bit and 2GB for 64bit systems
  • Click through the remaining wizard pages
  • Connect an ISO image (probably from fs2-qa1) and select "Connect at power on"
  • Start the VM and install the system + upgrades, and make configuration changes
  • Disconnect the ISO image
  • Install VMware tools
  • Shutdown the VM
  • Go to VM preferences and check time synchronization with the host under 'Options > VMware Tools'
  • Enable 'auto update' of VMware tools under 'Options > VMware Tools'
  • Create snapshot: "Initial State"
  • Right click the VM and select "Template > Convert to template"
Updating an existent template
  • Log into vsphere via the local client
  • Select "Inventory -> VMs and Templates"
  • Right click the template to modify and select "Convert to Virtual Machine..."
  • Start the VM
  • Perform all the necessary changes
  • Shutdown the VM
  • Create snapshot: e.g. "Update 01.07.2012"
  • Right click the VM and select "Template > Convert to template"

Deploying new machines

When a new machine is getting deployed from a template ensure the following tasks are getting executed:

  • Choose the template you want to deploy a new machine from
  • Select "Deploy a new virtual machine"
  • Select an unique name for the new machine, e.g. mm-win-7-32-1
  • Select the appropriate sub folder in the inventory location
  • Select the host for the VM (See the above host usage first)
  • Select the local disk for the destination storage of the VMware image
  • Click through and do no configuration
  • When finished start the machine
  • Change the hostname of the machine so it corresponds to above chosen machine name

Platform Requirements

For each of the supported platforms a VM has to be created or in case of OS X a Mac Mini to be installed. There are system and software requirements we have to fulfill when creating those systems. Therefore the following list will show what has to be performed until the VM/machine can be used for Mozmill and WebQA related automation tasks.

Linux (Ubuntu)
  • Create an user 'mozauto' which should auto-login
  • Install OpenSSH Server: sudo apt-get install openssh-server
  • Enable 'Desktop Sharing' for VNC connections by setting a password and disabling confirmation (see the tutorial)
  • Turn off screensaver, screen locking, and energy power settings
  • Put Terminal, and Software Upgrade application into the launcher
  • Install all the latest security fixes, and turn off automatic update checks, major version notifications, and set security updates to download automatically but not install
  • Install the latest version of Java JRE (see the [http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-ubuntu-via.html tutorial)
  • Install the latest version of Flash
Windows (All)
  • Create an user 'mozauto' in the admin group which should auto-login (use TweakUI for WinXP and 'netplwiz' for newer releases)
  • Connect network drive \\fs2.office.mozilla.org\qa1 as drive Z
  • Disable password expiration: z:\\tools\scripts\disable_password_expire.cmd
  • Install Ultra-VNC from Z:\setup\
  • Enable Remote Desktop for Windows XP and later
  • Set the hostname to match the os and version (e.g. windows_7_32) via the control panel
  • Turn off screensaver, screen locking, and energy power settings
  • Turn off UAC (user account control) for Windows Vista and upwards
  • Install all the latest security fixes, and turn off automatic updates
  • Install the latest version of Java JRE (Windows 2000 needs Java 6 Update 31)
  • Install the latest version of Flash (Windows 2000 needs Flash 10.1)
Mac OS X
  • Create an user 'mozauto' in the admin group which should auto-login
  • Enable Screen Sharing or better Remote Management and Remote Login
  • Add network connection: 'smb://fs2.private.mtv1.mozilla.com/qa1' (add to keychain)
  • Turn off screensaver, screen locking, and energy power settings
  • Enable automatic restart after power failure
  • Mute sound output
  • Install all the latest security fixes, and turn off automatic update checks
  • Install the latest version of Java JRE (type 'java' in the Terminal)
  • Install the latest version of Flash
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Infrastructure/Automation_Servers&oldid=462716"