QA/Execution/Web Testing/Automation/Jenkins

From MozillaWiki
Jump to navigation Jump to search

Summary

Jenkins CI is used by Web QA to run automated tests. The web dashboard can be accessed here (VPN required).

Installation

Jenkins has been installed using the native Mac OS X installer. It runs as a launch daemon, listening on port 8080. (If you want to know more about launchd and daemons, see here and here.)

Configuration

Boot configuration

The following have been configured for when Jenkins boots:

SETTING VALUE
heapSize 2g
permGen 512M
  • To view all settings in the file, run: defaults read /Library/Preferences/org.jenkins-ci
  • To get the value of a single setting, run: defaults read /Library/Preferences/org.jenkins-ci SETTING
  • To set the value of a setting, run: defaults write /Library/Preferences/org.jenkins-ci SETTING VALUE

Other configuration

  • Location of jenkins.war: /Applications/Jenkins/jenkins.war
  • Location of Jenkins home directory: /Users/Shared/Jenkins/Home

Starting/restarting/stopping the service

  • To restart via the dashboard visit http://qa-selenium.mv.mozilla.com:8080/restart
  • To manually start the daemon: sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
  • To manually stop the daemon: sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist

Plugins

Below is a list of plugins in use (dependencies have been omitted):

Global Environment Variables

Several environment variables are configured in the Configure System page:

  • DEFAULT_FIREFOX_VERSION - The default Firefox version to use. The default should be the latest compatible version.
  • PATH - The system PATH for binaries. The default is $PATH:/usr/local/bin.
  • PIP_DOWNLOAD_CACHE - Cache PIP downloads. The default is /Users/Shared/Jenkins/.pip_download_cache.
  • PIP_INDEX_URL - Sets the PyPI index URL. The default is http://pypi.build.mtv1.svc.mozilla.com/simple.
  • PIP_USE_MIRRORS - Controls use of the PyPI mirrors. By default this is false to speed up PIP installations.
  • PIP_VERBOSE - Controls the verbosity of PIP output. By default this is false and useful to enable for debugging.
  • PYTEST_PROCESSES_MAC - Number of parallel py.test processes to use on Mac nodes.
  • PYTEST_PROCESSES_SAUCE - Number of parallel py.test processes to use on Sauce Labs jobs.
  • PYTEST_PROCESSES_WINDOWS - Number of parallel py.test processes to use on Windows nodes.
  • PYTEST_RERUNS - Number of times to rerun a failed test (used by pytest-rerunfailures). The default is 2.
  • PYTEST_TIMEOUT - Timeout (in seconds) for each test (used by pytest-timeout). The default is 300 (five minutes).

Upgrading

When upgrades are available they can be automatically installed from the web dashboard.

Disaster recovery

Backing up

Jenkins is entirely file based so a true full backup would be simply storing a copy of the entire home directory in another location. That said, the ThinBackup plugin is used to perform scheduled (daily) backups of the Jenkins system and job configurations to http://fs2.office.mozilla.org/public/QA/WebQA/Jenkins/. Note that this backup does not include plugins.

Restoring from backup

If you are simply restoring system or job configuration then you can use the backups created by the ThinBackup plugin.

Migrating to a new server

If you are recovering from a hardware failure or are migrating to a new server then the backups created by the ThinBackup plugin will not be adequate. If you have access to the original server then the best way to restore is to simply copy the entire Jenkins home directory to the new server. Make sure that the instance on the original server is stopped before performing the copy.

If for some reason you do not have access to the original server then a partial backup will have to do. Install the ThinBackup plugin to a clean Jenkins instance and restore from the latest backup. Any plugins that are needed will have to be installed manually.

Console

Log messages can be seen by opening the Console application. A log query named Jenkins CI has been created, which filters by the sender: org.jenkins-ci.

Slaves

The following Jenkins slaves are configured:

unagi-1

  • Host: 10.250.1.52
  • Location: Mountain View
  • Devices: 1x Unagi
  • Role: UI Testing

This is a dual boot Mac Mini. On restarting if you are presented with DeployStudio you will need to power off and hold CTRL when switching on. This should boot the machine into Ubuntu.

unagi-2

  • Host: 10.250.120.75
  • Location: Haxxor, Mountain View
  • Devices: 1x Unagi
  • Role: Performance Testing

unagi-3

  • Host: 10.250.1.53
  • Location: Mountain View
  • Devices: 1x Unagi
  • Role: UI Testing

unagi-4

  • Host: 10.250.6.159
  • Location: Mountain View
  • Devices: 1x Unagi
  • Role: UI Testing

Troubleshooting

Upgrading

Automatic upgrade is failing or not available

This may be because the location of the jenkins.war file has the wrong permissions. These can be reset using the command: sudo chown -R daemon /Applications/Jenkins/

Backups

Cannot perform a backup

If you see the messages in the Jenkins log then it's likely the that remote volume for the backup destination is not mounted.

  • Cannot perform a backup. Please be sure jenkins/hudson has write privileges in the configured backup path {0}.
  • Could not create backup directory. Backup cannot be performed.

To fix this, connect to qa-selenium.mv.mozilla.com, and execute the following from a terminal (you can use SSH or VNC): 

   mkdir /Volumes/public
   sudo mount -t smbfs //guest@fs02/public /Volumes/public

Jobs

General issues

Here are a few tips that could help to diagnose the cause of general issues:

  • Enable verbose PIP output in the Configure System page to determine if installation of Python packages is causing issues.

PIP index is unavailable

We use an internal mirror of PyPI for installing Python packages. If this experiences issues then we can enable use of PIP mirrors in the Configure System page. Alternatively, temporarily change the PIP index URL.

PIP installation is slow

If Python packages are slow to install you should check that PIP mirrors are not enabled in the Configure System page. If mirrors are not enabled, it may be that the internal PyPI mirror is slow to respond. In this case, try temporarily changing the PIP index URL.

References

Some content has been taken from here.

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Execution/Web_Testing/Automation/Jenkins&oldid=512053"