QA/Execution/Web Testing/Automation/Jenkins

Summary

Jenkins CI is used by Web QA to run automated tests. The web dashboard can be accessed here (VPN required).

Installation

Jenkins has been installed using the native Mac OS X installer. It runs as a launch daemon, listening on port 8080. (If you want to know more about launchd and daemons, see here and here.)

Configuration

Boot configuration

The following have been configured for when Jenkins boots:

SETTING	VALUE
heapSize	2g
permGen	512M

• To view all settings in the file, run: defaults read /Library/Preferences/org.jenkins-ci
• To get the value of a single setting, run: defaults read /Library/Preferences/org.jenkins-ci SETTING
• To set the value of a setting, run: defaults write /Library/Preferences/org.jenkins-ci SETTING VALUE

LDAP authentication

We use LDAP Auth coupled with job-execution and job/system-config changes, so we know who did what, for easier tracking/troubleshooting

To set it up:

1. Load http://qa-selenium.mv.mozilla.com:8080/configureSecurity/
2. Click the checkbox to "Enable Security"
3. Under "Access Control," select "LDAP" for the Security Realm
4. Enter the appropriate settings (see bug 904287 for the values)
5. Now, choose "Logged-in users can do anything" from Authorization

Other configuration

• Location of jenkins.war: /Applications/Jenkins/jenkins.war
• Location of Jenkins home directory: /Users/Shared/Jenkins/Home

Starting/restarting/stopping the service

• To restart via the dashboard visit http://qa-selenium.mv.mozilla.com:8080/restart
• To manually start the daemon: sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
• To manually stop the daemon: sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist

Plugins

Below is a list of plugins in use (dependencies have been omitted):

• LDAP - User authentication
• IRC - IRC bot for triggering builds and notifications
• EnvInject - Set environment variables for jobs
• URLTrigger - Trigger builds based on the response of a URL
• Build Timeout - Abort builds that take too long
• github - Integrates github
• Green Balls - Green is better than blue!
• HTML Publisher - Publishes HTML reports
• Jenkins Job Configuration History - Provides history of configuration changes
• ThinBackup - Scheduled backups of system and job configurations
• xUnit - Generates reports based on test results
• SSH Slaves - Allows you to manage slaves running on *nix machines over SSH
• Mask Passwords - Allows masking passwords that may appear in the console
• Multiple SCMs - Allows a job to check out sources from multiple SCM providers
• Timestamper - Adds timestamps to the console output
• Rebuild - Allows the user to rebuild a parametrized build without entering the parameters again
• Workspace Cleanup - Deletes the workspace before the build or when a build is finished and artifacts saved
• AnsiColor - Adds support for ANSI escape sequences in console output

Global Environment Variables

Several environment variables are configured in the Configure System page:

• DEFAULT_FIREFOX_VERSION - The default Firefox version to use. The default should be the latest compatible version.
• PATH - The system PATH for binaries. The default is $PATH:/usr/local/bin.
• PIP_DOWNLOAD_CACHE - Cache PIP downloads. The default is /Users/Shared/Jenkins/.pip_download_cache.
• PIP_INDEX_URL - Sets the PyPI index URL. The default is empty, to use the main PyPI index.
• PIP_USE_MIRRORS - Controls use of the PyPI mirrors. By default this is false to speed up PIP installations.
• PIP_VERBOSE - Controls the verbosity of PIP output. By default this is false and useful to enable for debugging.
• PYTEST_PROCESSES_MAC - Number of parallel py.test processes to use on Mac nodes.
• PYTEST_PROCESSES_SAUCE - Number of parallel py.test processes to use on Sauce Labs jobs.
• PYTEST_PROCESSES_WINDOWS - Number of parallel py.test processes to use on Windows nodes.
• PYTEST_RERUNS - Number of times to rerun a failed test (used by pytest-rerunfailures). The default is 2.
• PYTEST_TIMEOUT - Timeout (in seconds) for each test (used by pytest-timeout). The default is 300 (five minutes).

B2G Nodes

• See: B2G Nodes

Upgrading

When upgrades are available they can be automatically installed from the web dashboard.

Disaster recovery

Backing up

Jenkins is entirely file based so a true full backup would be simply storing a copy of the entire home directory in another location. That said, the ThinBackup plugin is used to perform scheduled (daily) backups of the Jenkins system and job configurations to http://fs2.office.mozilla.org/public/QA/WebQA/Jenkins/. Note that this backup does not include plugins.

Restoring from backup

If you are simply restoring system or job configuration then you can use the backups created by the ThinBackup plugin.

Migrating to a new server

If you are recovering from a hardware failure or are migrating to a new server then the backups created by the ThinBackup plugin will not be adequate. If you have access to the original server then the best way to restore is to simply copy the entire Jenkins home directory to the new server. Make sure that the instance on the original server is stopped before performing the copy.

If for some reason you do not have access to the original server then a partial backup will have to do. Install the ThinBackup plugin to a clean Jenkins instance and restore from the latest backup. Any plugins that are needed will have to be installed manually.

Console

Log messages can be seen by opening the Console application. A log query named Jenkins CI has been created, which filters by the sender: org.jenkins-ci.

Troubleshooting

Upgrading

Automatic upgrade is failing or not available

This may be because the location of the jenkins.war file has the wrong permissions. These can be reset using:

   sudo chown -R daemon /Applications/Jenkins/
   sudo chmod a=rw /Applications/Jenkins/jenkins.*

Backups

Cannot perform a backup

If you see the messages in the Jenkins log then it's likely the that remote volume for the backup destination is not mounted.

• Cannot perform a backup. Please be sure jenkins/hudson has write privileges in the configured backup path {0}.
• Could not create backup directory. Backup cannot be performed.

To fix this, connect to qa-selenium.mv.mozilla.com, and execute the following from a terminal (you can use SSH or VNC):

   mkdir /Volumes/public
   sudo mount -t smbfs //guest@fs02/public /Volumes/public

Jobs

General issues

Here are a few tips that could help to diagnose the cause of general issues:

• Enable verbose PIP output in the Configure System page to determine if installation of Python packages is causing issues.

PIP index is unavailable

We use the main PyPI index for installing Python packages. If this experiences issues then we can enable use of PIP mirrors in the Configure System page. Alternatively, we can temporarily change the PIP index URL to our internal mirror: http://pypi.build.mtv1.svc.mozilla.com/simple.

PIP installation is slow

If Python packages are slow to install you should check that PIP mirrors are not enabled in the Configure System page. If mirrors are not enabled, it may be that the main PyPI mirror is slow to respond. In this case, try temporarily changing the PIP index URL to the internal mirror: http://pypi.build.mtv1.svc.mozilla.com/simple.

Stalled node jobs

If jobs running on nodes appear to have stalled in post-build actions it may be that the node momentarily lost connection with the master. We have seen this with our Linux nodes running Open JDK. The problem was resolved by switching to Oracle's JDK. To do this, take the node temporarily offline and SSH into the box. Then run the following commands to remove Open JDK and install Oracle's JDK:

   sudo apt-get purge openjdk*
   sudo add-apt-repository ppa:webupd8team/java
   sudo apt-get update
   sudo apt-get install oracle-java7-installer

You can confirm the version of Java using the command:

   java -version

Once this is done, you can launch the node again from Jenkins web console.

References

Some content has been taken from here.