QA/Execution/Web Testing/Automation/Jenkins: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(48 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= Summary =
= Summary =
Jenkins CI is used by Web QA to run automated tests. The web dashboard can be accessed [http://qa-selenium.mv.mozilla.com:8080 here] (VPN required).
[http://jenkins-ci.org/ Jenkins CI] is used by Web QA to run automated tests. The production instance is publicly accessible and can be found [https://webqa-ci.mozilla.com/ here]. Details of the setup and administration are available on our [https://mana.mozilla.org/wiki/display/QA/webqa-ci.mozilla.com internal Mana page]. We also have a staging instance for adhoc jobs and experimentation. This is secured behind Mozilla VPN, and can be found [http://webqa-ci-staging1.qa.scl3.mozilla.com:8080/ here]. Documentation for the staging instance is also available [https://mana.mozilla.org/wiki/display/QA/webqa-ci-staging1.qa.scl3.mozilla.com on Mana].
 
= Installation =
Jenkins has been installed using the [http://mirrors.jenkins-ci.org/osx/latest native Mac OS X installer]. It runs as a launch daemon, listening on port 8080. (If you want to know more about launchd and daemons, see [http://developer.apple.com/library/mac/#technotes/tn2083/_index.html here] and [http://developer.apple.com/library/mac/#documentation/MacOSX/Conceptual/BPSystemStartup/Articles/LaunchOnDemandDaemons.html here].)
 
= Configuration =
 
== Boot configuration ==
The following have been configured for when Jenkins boots:
 
{|
! SETTING !! VALUE
|-
| heapSize || 2g
|-
| permGen || 512M
|}
 
* To view all settings in the file, run: <code>defaults read /Library/Preferences/org.jenkins-ci</code>
* To get the value of a single setting, run: <code>defaults read /Library/Preferences/org.jenkins-ci SETTING</code>
* To set the value of a setting, run: <code>defaults write /Library/Preferences/org.jenkins-ci SETTING VALUE</code>
 
== Other configuration ==
* Location of jenkins.war: /Applications/Jenkins/jenkins.war
* Location of Jenkins home directory: /Users/Shared/Jenkins/Home
 
= Starting/restarting/stopping the service =
* To restart via the dashboard visit [http://qa-selenium.mv.mozilla.com:8080/restart http://qa-selenium.mv.mozilla.com:8080/restart]
* To manually start the daemon: <code>sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist</code>
* To manually stop the daemon: <code>sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist</code>


= Plugins =
= Plugins =
Below is a list of plugins in use (dependencies have been omitted):
Below is a list of plugins in use (dependencies have been omitted):
 
* [https://wiki.jenkins-ci.org/display/JENKINS/Build+Failure+Analyzer Build Failure Analyzer] - Helps collate build failures
* [http://wiki.jenkins-ci.org/display/JENKINS/LDAP+Plugin LDAP] - User authentication
* [https://wiki.jenkins-ci.org/display/JENKINS/Build+Monitor+Plugin Build Monitor] - Highly visible view of the status of selected jobs
* [http://wiki.jenkins-ci.org/display/JENKINS/IRC+Plugin IRC] - IRC bot for triggering builds and notifications
* [http://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin Build Timeout] - Allows users to take responsibility for failed builds
* [https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin Claim] - Ability to assign build failures
* [https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin Config File Provider] - Centrally maintained configuration files for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View Dashboard View] - Dashboard view for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin EnvInject] - Set environment variables for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin EnvInject] - Set environment variables for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/URLTrigger+Plugin URLTrigger] - Trigger builds based on the response of a URL
* [http://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin GitHub] - Integrates GitHub
* [http://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin Build Timeout] - Abort builds that take too long
* [http://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin github] - Integrates github
* [http://wiki.jenkins-ci.org/display/JENKINS/Green+Balls Green Balls] - Green is better than blue!
* [http://wiki.jenkins-ci.org/display/JENKINS/Green+Balls Green Balls] - Green is better than blue!
* [http://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin HTML Publisher] - Publishes HTML reports
* [http://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin HTML Publisher] - Publishes HTML reports
* [http://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin Jenkins Job Configuration History] - Provides history of configuration changes
* [http://wiki.jenkins-ci.org/display/JENKINS/IRC+Plugin IRC] - IRC bot for triggering builds and notifications
* [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup] - Scheduled backups of system and job configurations
* [http://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin Job Configuration History] - Provides history of configuration changes
* [http://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin xUnit] - Generates reports based on test results
* [http://wiki.jenkins-ci.org/display/JENKINS/LDAP+Plugin LDAP] - User authentication
* [https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin SSH Slaves] - Allows you to manage slaves running on *nix machines over SSH
* [https://wiki.jenkins-ci.org/display/JENKINS/Managed+Script+Plugin Managed Script] - Centrally maintained scripts for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/Mask+Passwords+Plugin Mask Passwords] - Allows masking passwords that may appear in the console
* [https://wiki.jenkins-ci.org/display/JENKINS/Monitoring Monitoring] - Monitoring of Jenkins itself
* [https://wiki.jenkins-ci.org/display/JENKINS/Multiple+SCMs+Plugin Multiple SCMs] - Allows a job to check out sources from multiple SCM providers
* [https://wiki.jenkins-ci.org/display/JENKINS/Ownership+Plugin Ownership] - Show job owners and co-owners
* [https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin JUnit Attachments] - Archive certain files (attachments) together with JUnit results
* [https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin Parameterized Trigger] - Trigger builds with parameters
* [https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin Rebuild] - Rebuild a parametrized build without re-entering the parameters
* [https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Plugin Role Strategy] - Role-based strategy for user management
* [https://github.com/mozilla/testdroid-marionette-plugin Testdroid Marionette] - Provisions a Firefox OS device via Testdroid
* [https://wiki.jenkins-ci.org/display/JENKINS/Test+stability+plugin Test Stability] - Historical information about the stability of tests
* [https://wiki.jenkins-ci.org/display/JENKINS/Timestamper Timestamper] - Adds timestamps to the console output
* [https://wiki.jenkins-ci.org/display/JENKINS/Timestamper Timestamper] - Adds timestamps to the console output
* [https://wiki.jenkins-ci.org/display/JENKINS/URLTrigger+Plugin URLTrigger] - Trigger builds based on the response of a URL
* [https://wiki.jenkins-ci.org/display/JENKINS/View+Job+Filters View Job Filters] - Advanced filters for views
* [https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin Workspace Cleanup] - Deletes the workspace before or after the build


= Global Environment Variables =
= Global Environment Variables =
Several environment variables are configured in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page:
Several environment variables are configured:


* '''DEFAULT_FIREFOX_VERSION''' - The default Firefox version to use. The default should be the latest compatible version.
* '''APPIUM_BROWSER_NAME''' - Browser name to use with Appium. The default is '''Browser'''.
* '''PATH''' - The system PATH for binaries. The default is '''$PATH:/usr/local/bin'''
* '''APPIUM_DEVICE_NAME''' - Device name to use with Appium. The default is '''Android Emulator'''.
* '''PIP_DOWNLOAD_CACHE''' - Cache PIP downloads. The default is '''/Users/Shared/Jenkins/.pip_download_cache'''
* '''APPIUM_PLATFORM_NAME''' - Platform name to use with Appium. The default is '''Android'''.
* '''PIP_INDEX_URL''' - Sets the PyPI index URL. The default is '''<nowiki>http://pypi.build.mtv1.svc.mozilla.com/simple</nowiki>'''
* '''APPIUM_PLATFORM_VERSION''' - Platform version to use with Appium. The default is '''5.1'''.
* '''APPIUM_VERSION''' - Version of Appium to use. The default is '''1.5.3'''.
* '''FIREFOX_VERSION''' - Firefox version to use. The default is '''47.0'''.
* '''MAC_VERSION''' - MacOS version to use. The default is '''OS X 10.11'''.
* '''PIP_INDEX_URL''' - Sets the PyPI index URL. The default is empty, to use the main PyPI index.
* '''PIP_USE_MIRRORS''' - Controls use of the PyPI mirrors. By default this is '''false''' to speed up PIP installations.
* '''PIP_USE_MIRRORS''' - Controls use of the PyPI mirrors. By default this is '''false''' to speed up PIP installations.
* '''PIP_VERBOSE''' - Controls the verbosity of PIP output. By default this is '''false''' and useful to enable for debugging.
* '''PIP_VERBOSE''' - Controls the verbosity of PIP output. By default this is '''false''' and useful to enable for debugging.
* '''PYTEST_PROCESSES_MAC''' - Number of parallel py.test processes to use on Mac nodes.
* '''PYTEST_ADDOPTS''' - Add py.test options to all jobs. The default is '''-r=fsxXR --verbose --tb=short'''.
* '''PYTEST_PROCESSES_SAUCE''' - Number of parallel py.test processes to use on [http://saucelabs.com/ Sauce Labs] jobs.
* '''PYTEST_PROCESSES''' - Number of parallel py.test processes to use. The default is '''15'''.
* '''PYTEST_PROCESSES_WINDOWS''' - Number of parallel py.test processes to use on Windows nodes.
* '''PYTEST_RERUNS''' - Number of times to rerun a failed test (used by [https://pypi.python.org/pypi/pytest-rerunfailures pytest-rerunfailures]). The default is '''5'''.
* '''PYTEST_RERUNS''' - Number of times to rerun a failed test (used by pytest-rerunfailures).
* '''SAUCELABS_USERNAME''' - Username to use for [http://saucelabs.com/ Sauce Labs].
 
* '''SELENIUM_EXCLUDE_DEBUG''' - Exclude capturing these debug types from Selenium tests. The default is '''logs'''.
= Upgrading =
* '''SELENIUM_VERSION''' - Selenium version to use. The default is '''2.53.1'''.
When upgrades are available they can be automatically installed from the [http://qa-selenium.mv.mozilla.com:8080/manage web dashboard].
* '''TOX_TESTENV_PASSENV''' - Environment variables to pass down into Tox environments. The default is '''http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY SAUCELABS_USERNAME SAUCELABS_API_KEY'''.
 
* '''WINDOWS_VERSION''' - Windows version to use. The default is '''Windows 7'''.
= Disaster recovery =
 
== Backing up ==
Jenkins is entirely file based so a true full backup would be simply storing a copy of the entire home directory in another location. That said, the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin] is used to perform scheduled (daily) backups of the Jenkins system and job configurations to [http://fs2.office.mozilla.org/public/QA/WebQA/Jenkins/ http://fs2.office.mozilla.org/public/QA/WebQA/Jenkins/]. Note that this backup does '''not''' include plugins.
 
== Restoring from backup ==
If you are simply restoring system or job configuration then you can use the backups created by the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin].


== Migrating to a new server ==
= Known Issues =
If you are recovering from a hardware failure or are migrating to a new server then the backups created by the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin] will not be adequate. If you have access to the original server then the best way to restore is to simply copy the entire Jenkins home directory to the new server. Make sure that the instance on the original server is stopped before performing the copy.
* None
 
If for some reason you do not have access to the original server then a partial backup will have to do. Install the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin] to a clean Jenkins instance and restore from the latest backup. Any plugins that are needed will have to be installed manually.
 
= Console =
Log messages can be seen by opening the Console application. A log query named Jenkins CI has been created, which filters by the sender: <code>org.jenkins-ci</code>.
 
= Slaves =
The following Jenkins slaves are configured:
 
== unagi-1 (offline) ==
* Host: 10.250.1.52
* Location: Mountain View
* Devices: 1x Unagi
 
This is a dual boot Mac Mini. On restarting if you are presented with DeployStudio ypu will need to power off and hold CTRL when switching on. This should boot the machine into Ubuntu.
 
== unagi-2 ==
* Host: 10.250.120.75
* Location: Haxxor, Mountain View
* Devices: 1x Unagi
 
== unagi-3 (offline) ==
* Host: 10.250.1.53
* Location: Mountain View
* Devices: 1x Unagi


= Troubleshooting =
= Troubleshooting =
Try enabling verbose PIP output to determine if installation of Python packages is causing issues.


== Upgrading ==
== PIP index is unavailable ==
 
We use the main [https://pypi.python.org/pypi PyPI] (Python Package Index) for installing Python packages. If this experiences issues then we can enable use of PIP mirrors.
=== Automatic upgrade is failing or not available ===
This may be because the location of the jenkins.war file has the wrong permissions. These can be reset using the command: <code>sudo chown -R daemon /Applications/Jenkins/</code>
 
== Backups ==
 
=== Cannot perform a backup ===
If you see the messages in the Jenkins log then it's likely the that remote volume for the backup destination is not mounted.
* Cannot perform a backup. Please be sure jenkins/hudson has write privileges in the configured backup path {0}.
* Could not create backup directory. Backup cannot be performed.
 
To fix this, connect to qa-selenium.mv.mozilla.com, and execute the following from a terminal (you can use SSH or VNC):
 
    mkdir /Volumes/public
    sudo mount -t smbfs //guest@fs02/public /Volumes/public
 
== Jobs ==
 
=== General issues ===
Here are a few tips that could help to diagnose the cause of general issues:
* Enable verbose PIP output in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page to determine if installation of Python packages is causing issues.
 
=== PIP index is unavailable ===
We use an internal mirror of PyPI for installing Python packages. If this experiences issues then we can enable use of PIP mirrors in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page. Alternatively, temporarily change the PIP index URL.
 
=== PIP installation is slow ===
If Python packages are slow to install you should check that PIP mirrors are not enabled in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page. If mirrors are not enabled, it may be that the internal PyPI mirror is slow to respond. In this case, try temporarily changing the PIP index URL.


= References =
== PIP installation is slow ==
Some content has been taken from [https://wiki.jenkins-ci.org/display/JENKINS/Thanks+for+using+OSX+Installer here].
If Python packages are slow to install you should check that PIP mirrors are not enabled. If not, it may be that the main PyPI mirror is slow to respond.

Latest revision as of 13:22, 6 October 2016

Summary

Jenkins CI is used by Web QA to run automated tests. The production instance is publicly accessible and can be found here. Details of the setup and administration are available on our internal Mana page. We also have a staging instance for adhoc jobs and experimentation. This is secured behind Mozilla VPN, and can be found here. Documentation for the staging instance is also available on Mana.

Plugins

Below is a list of plugins in use (dependencies have been omitted):

Global Environment Variables

Several environment variables are configured:

  • APPIUM_BROWSER_NAME - Browser name to use with Appium. The default is Browser.
  • APPIUM_DEVICE_NAME - Device name to use with Appium. The default is Android Emulator.
  • APPIUM_PLATFORM_NAME - Platform name to use with Appium. The default is Android.
  • APPIUM_PLATFORM_VERSION - Platform version to use with Appium. The default is 5.1.
  • APPIUM_VERSION - Version of Appium to use. The default is 1.5.3.
  • FIREFOX_VERSION - Firefox version to use. The default is 47.0.
  • MAC_VERSION - MacOS version to use. The default is OS X 10.11.
  • PIP_INDEX_URL - Sets the PyPI index URL. The default is empty, to use the main PyPI index.
  • PIP_USE_MIRRORS - Controls use of the PyPI mirrors. By default this is false to speed up PIP installations.
  • PIP_VERBOSE - Controls the verbosity of PIP output. By default this is false and useful to enable for debugging.
  • PYTEST_ADDOPTS - Add py.test options to all jobs. The default is -r=fsxXR --verbose --tb=short.
  • PYTEST_PROCESSES - Number of parallel py.test processes to use. The default is 15.
  • PYTEST_RERUNS - Number of times to rerun a failed test (used by pytest-rerunfailures). The default is 5.
  • SAUCELABS_USERNAME - Username to use for Sauce Labs.
  • SELENIUM_EXCLUDE_DEBUG - Exclude capturing these debug types from Selenium tests. The default is logs.
  • SELENIUM_VERSION - Selenium version to use. The default is 2.53.1.
  • TOX_TESTENV_PASSENV - Environment variables to pass down into Tox environments. The default is http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY SAUCELABS_USERNAME SAUCELABS_API_KEY.
  • WINDOWS_VERSION - Windows version to use. The default is Windows 7.

Known Issues

  • None

Troubleshooting

Try enabling verbose PIP output to determine if installation of Python packages is causing issues.

PIP index is unavailable

We use the main PyPI (Python Package Index) for installing Python packages. If this experiences issues then we can enable use of PIP mirrors.

PIP installation is slow

If Python packages are slow to install you should check that PIP mirrors are not enabled. If not, it may be that the main PyPI mirror is slow to respond.

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Execution/Web_Testing/Automation/Jenkins&oldid=1150443"