QA/Execution/Web Testing/Automation/Jenkins: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(30 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= Summary =
= Summary =
Jenkins CI is used by Web QA to run automated tests. The web dashboard can be accessed [http://qa-selenium.mv.mozilla.com:8080 here] (VPN required).
[http://jenkins-ci.org/ Jenkins CI] is used by Web QA to run automated tests. The production instance is publicly accessible and can be found [https://webqa-ci.mozilla.com/ here]. Details of the setup and administration are available on our [https://mana.mozilla.org/wiki/display/QA/webqa-ci.mozilla.com internal Mana page]. We also have a staging instance for adhoc jobs and experimentation. This is secured behind Mozilla VPN, and can be found [http://webqa-ci-staging1.qa.scl3.mozilla.com:8080/ here]. Documentation for the staging instance is also available [https://mana.mozilla.org/wiki/display/QA/webqa-ci-staging1.qa.scl3.mozilla.com on Mana].
 
= Installation =
Jenkins has been installed using the [http://mirrors.jenkins-ci.org/osx-stable/latest native Mac OS X installer]. It runs as a launch daemon, listening on port 8080. (If you want to know more about launchd and daemons, see [http://developer.apple.com/library/mac/#technotes/tn2083/_index.html here] and [http://developer.apple.com/library/mac/#documentation/MacOSX/Conceptual/BPSystemStartup/Articles/LaunchOnDemandDaemons.html here].)
 
= Configuration =
 
== Boot configuration ==
The following have been configured for when Jenkins boots:
 
{|
! SETTING !! VALUE
|-
| heapSize || 2g
|-
| permGen || 512M
|}
 
* To view all settings in the file, run: <code>defaults read /Library/Preferences/org.jenkins-ci</code>
* To get the value of a single setting, run: <code>defaults read /Library/Preferences/org.jenkins-ci SETTING</code>
* To set the value of a setting, run: <code>defaults write /Library/Preferences/org.jenkins-ci SETTING VALUE</code>
 
== LDAP authentication ==
 
We use LDAP Auth coupled with job-execution and job/system-config changes, so we know who did what, for easier tracking/troubleshooting
 
To set it up:
# Load http://qa-selenium.mv.mozilla.com:8080/configureSecurity/
# Click the checkbox to "Enable Security"
# Under "Access Control," select "LDAP" for the Security Realm
# Enter the appropriate settings (see [https://bugzilla.mozilla.org/show_bug.cgi?id=904287 bug 904287] for the values)
# Now, choose "Logged-in users can do anything" from Authorization
 
== Other configuration ==
* Location of jenkins.war: /Applications/Jenkins/jenkins.war
* Location of Jenkins home directory: /Users/Shared/Jenkins/Home
 
= Starting/restarting/stopping the service =
* To restart via the dashboard visit [http://qa-selenium.mv.mozilla.com:8080/restart http://qa-selenium.mv.mozilla.com:8080/restart]
* To manually start the daemon: <code>sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist</code>
* To manually stop the daemon: <code>sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist</code>


= Plugins =
= Plugins =
Below is a list of plugins in use (dependencies have been omitted):
Below is a list of plugins in use (dependencies have been omitted):
 
* [https://wiki.jenkins-ci.org/display/JENKINS/Build+Failure+Analyzer Build Failure Analyzer] - Helps collate build failures
* [http://wiki.jenkins-ci.org/display/JENKINS/LDAP+Plugin LDAP] - User authentication
* [https://wiki.jenkins-ci.org/display/JENKINS/Build+Monitor+Plugin Build Monitor] - Highly visible view of the status of selected jobs
* [http://wiki.jenkins-ci.org/display/JENKINS/IRC+Plugin IRC] - IRC bot for triggering builds and notifications
* [http://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin Build Timeout] - Allows users to take responsibility for failed builds
* [https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin Claim] - Ability to assign build failures
* [https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin Config File Provider] - Centrally maintained configuration files for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View Dashboard View] - Dashboard view for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin EnvInject] - Set environment variables for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin EnvInject] - Set environment variables for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/URLTrigger+Plugin URLTrigger] - Trigger builds based on the response of a URL
* [http://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin GitHub] - Integrates GitHub
* [http://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin Build Timeout] - Abort builds that take too long
* [http://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin github] - Integrates github
* [http://wiki.jenkins-ci.org/display/JENKINS/Green+Balls Green Balls] - Green is better than blue!
* [http://wiki.jenkins-ci.org/display/JENKINS/Green+Balls Green Balls] - Green is better than blue!
* [http://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin HTML Publisher] - Publishes HTML reports
* [http://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin HTML Publisher] - Publishes HTML reports
* [http://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin Jenkins Job Configuration History] - Provides history of configuration changes
* [http://wiki.jenkins-ci.org/display/JENKINS/IRC+Plugin IRC] - IRC bot for triggering builds and notifications
* [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup] - Scheduled backups of system and job configurations
* [http://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin Job Configuration History] - Provides history of configuration changes
* [http://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin xUnit] - Generates reports based on test results
* [http://wiki.jenkins-ci.org/display/JENKINS/LDAP+Plugin LDAP] - User authentication
* [https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin SSH Slaves] - Allows you to manage slaves running on *nix machines over SSH
* [https://wiki.jenkins-ci.org/display/JENKINS/Managed+Script+Plugin Managed Script] - Centrally maintained scripts for jobs
* [https://wiki.jenkins-ci.org/display/JENKINS/Mask+Passwords+Plugin Mask Passwords] - Allows masking passwords that may appear in the console
* [https://wiki.jenkins-ci.org/display/JENKINS/Monitoring Monitoring] - Monitoring of Jenkins itself
* [https://wiki.jenkins-ci.org/display/JENKINS/Multiple+SCMs+Plugin Multiple SCMs] - Allows a job to check out sources from multiple SCM providers
* [https://wiki.jenkins-ci.org/display/JENKINS/Ownership+Plugin Ownership] - Show job owners and co-owners
* [https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin Parameterized Trigger] - Trigger builds with parameters
* [https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin Rebuild] - Rebuild a parametrized build without re-entering the parameters
* [https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Plugin Role Strategy] - Role-based strategy for user management
* [https://github.com/mozilla/testdroid-marionette-plugin Testdroid Marionette] - Provisions a Firefox OS device via Testdroid
* [https://wiki.jenkins-ci.org/display/JENKINS/Test+stability+plugin Test Stability] - Historical information about the stability of tests
* [https://wiki.jenkins-ci.org/display/JENKINS/Timestamper Timestamper] - Adds timestamps to the console output
* [https://wiki.jenkins-ci.org/display/JENKINS/Timestamper Timestamper] - Adds timestamps to the console output
* [https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin Rebuild] - Allows the user to rebuild a parametrized build without entering the parameters again
* [https://wiki.jenkins-ci.org/display/JENKINS/URLTrigger+Plugin URLTrigger] - Trigger builds based on the response of a URL
* [https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin Workspace Cleanup] - Deletes the workspace before the build or when a build is finished and artifacts saved
* [https://wiki.jenkins-ci.org/display/JENKINS/View+Job+Filters View Job Filters] - Advanced filters for views
* [https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin Workspace Cleanup] - Deletes the workspace before or after the build


= Global Environment Variables =
= Global Environment Variables =
Several environment variables are configured in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page:
Several environment variables are configured:


* '''DEFAULT_FIREFOX_VERSION''' - The default Firefox version to use. The default should be the latest compatible version.
* '''APPIUM_BROWSER_NAME''' - Browser name to use with Appium. The default is '''Browser'''.
* '''PATH''' - The system PATH for binaries. The default is '''$PATH:/usr/local/bin'''.
* '''APPIUM_DEVICE_NAME''' - Device name to use with Appium. The default is '''Android Emulator'''.
* '''PIP_DOWNLOAD_CACHE''' - Cache PIP downloads. The default is '''/Users/Shared/Jenkins/.pip_download_cache'''.
* '''APPIUM_PLATFORM_NAME''' - Platform name to use with Appium. The default is '''Android'''.
* '''APPIUM_PLATFORM_VERSION''' - Platform version to use with Appium. The default is '''5.1'''.
* '''APPIUM_VERSION''' - Version of Appium to use. The default is '''1.5.3'''.
* '''FIREFOX_VERSION''' - Firefox version to use. The default is '''47.0'''.
* '''MAC_VERSION''' - MacOS version to use. The default is '''OS X 10.11'''.
* '''PIP_INDEX_URL''' - Sets the PyPI index URL. The default is empty, to use the main PyPI index.
* '''PIP_INDEX_URL''' - Sets the PyPI index URL. The default is empty, to use the main PyPI index.
* '''PIP_USE_MIRRORS''' - Controls use of the PyPI mirrors. By default this is '''false''' to speed up PIP installations.
* '''PIP_USE_MIRRORS''' - Controls use of the PyPI mirrors. By default this is '''false''' to speed up PIP installations.
* '''PIP_VERBOSE''' - Controls the verbosity of PIP output. By default this is '''false''' and useful to enable for debugging.
* '''PIP_VERBOSE''' - Controls the verbosity of PIP output. By default this is '''false''' and useful to enable for debugging.
* '''PYTEST_PROCESSES_MAC''' - Number of parallel py.test processes to use on Mac nodes.
* '''PYTEST_ADDOPTS''' - Add py.test options to all jobs. The default is '''-r=fsxXR --verbose --tb=short'''.
* '''PYTEST_PROCESSES_SAUCE''' - Number of parallel py.test processes to use on [http://saucelabs.com/ Sauce Labs] jobs.
* '''PYTEST_PROCESSES''' - Number of parallel py.test processes to use. The default is '''15'''.
* '''PYTEST_PROCESSES_WINDOWS''' - Number of parallel py.test processes to use on Windows nodes.
* '''PYTEST_RERUNS''' - Number of times to rerun a failed test (used by [https://pypi.python.org/pypi/pytest-rerunfailures pytest-rerunfailures]). The default is '''5'''.
* '''PYTEST_RERUNS''' - Number of times to rerun a failed test (used by pytest-rerunfailures). The default is '''2'''.
* '''SAUCELABS_USERNAME''' - Username to use for [http://saucelabs.com/ Sauce Labs].
* '''PYTEST_TIMEOUT''' - Timeout (in seconds) for each test (used by pytest-timeout). The default is '''300''' (five minutes).
* '''SELENIUM_EXCLUDE_DEBUG''' - Exclude capturing these debug types from Selenium tests. The default is '''logs'''.
* '''SELENIUM_VERSION''' - Selenium version to use. The default is '''2.53.1'''.
* '''TOX_TESTENV_PASSENV''' - Environment variables to pass down into Tox environments. The default is '''http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY SAUCELABS_USERNAME SAUCELABS_API_KEY'''.
* '''WINDOWS_VERSION''' - Windows version to use. The default is '''Windows 7'''.


= B2G Nodes =
= Known Issues =
* See: [[QA/Execution/Web_Testing/Automation/B2G_Nodes|B2G Nodes]]
* None
 
= Upgrading =
When upgrades are available they can be automatically installed from the [http://qa-selenium.mv.mozilla.com:8080/manage web dashboard].
 
= Disaster recovery =
 
== Backing up ==
Jenkins is entirely file based so a true full backup would be simply storing a copy of the entire home directory in another location. That said, the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin] is used to perform scheduled (daily) backups of the Jenkins system and job configurations to [http://fs2.office.mozilla.org/public/QA/WebQA/Jenkins/ http://fs2.office.mozilla.org/public/QA/WebQA/Jenkins/]. Note that this backup does '''not''' include plugins.
 
== Restoring from backup ==
If you are simply restoring system or job configuration then you can use the backups created by the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin].
 
== Migrating to a new server ==
If you are recovering from a hardware failure or are migrating to a new server then the backups created by the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin] will not be adequate. If you have access to the original server then the best way to restore is to simply copy the entire Jenkins home directory to the new server. Make sure that the instance on the original server is stopped before performing the copy.
 
If for some reason you do not have access to the original server then a partial backup will have to do. Install the [https://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackup plugin] to a clean Jenkins instance and restore from the latest backup. Any plugins that are needed will have to be installed manually.
 
= Console =
Log messages can be seen by opening the Console application. A log query named Jenkins CI has been created, which filters by the sender: <code>org.jenkins-ci</code>.


= Troubleshooting =
= Troubleshooting =
Try enabling verbose PIP output to determine if installation of Python packages is causing issues.


== Upgrading ==
== PIP index is unavailable ==
 
We use the main [https://pypi.python.org/pypi PyPI] (Python Package Index) for installing Python packages. If this experiences issues then we can enable use of PIP mirrors.
=== Automatic upgrade is failing or not available ===
This may be because the location of the jenkins.war file has the wrong permissions. These can be reset using:
 
    sudo chown -R daemon /Applications/Jenkins/
    sudo chmod a=rw /Applications/Jenkins/jenkins.*
 
== Backups ==
 
=== Cannot perform a backup ===
If you see the messages in the Jenkins log then it's likely the that remote volume for the backup destination is not mounted.
* Cannot perform a backup. Please be sure jenkins/hudson has write privileges in the configured backup path {0}.
* Could not create backup directory. Backup cannot be performed.
 
To fix this, connect to qa-selenium.mv.mozilla.com, and execute the following from a terminal (you can use SSH or VNC):
 
    mkdir /Volumes/public
    sudo mount -t smbfs //guest@fs02/public /Volumes/public
 
== Jobs ==
 
=== General issues ===
Here are a few tips that could help to diagnose the cause of general issues:
* Enable verbose PIP output in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page to determine if installation of Python packages is causing issues.
 
=== PIP index is unavailable ===
We use the main PyPI index for installing Python packages. If this experiences issues then we can enable use of PIP mirrors in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page. Alternatively, we can temporarily change the PIP index URL to our internal mirror: '''<nowiki>http://pypi.build.mtv1.svc.mozilla.com/simple</nowiki>'''.
 
=== PIP installation is slow ===
If Python packages are slow to install you should check that PIP mirrors are not enabled in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page. If mirrors are not enabled, it may be that the main PyPI mirror is slow to respond. In this case, try temporarily changing the PIP index URL to the internal mirror: '''<nowiki>http://pypi.build.mtv1.svc.mozilla.com/simple</nowiki>'''.
 
=== Stalled node jobs ===
If jobs running on nodes appear to have stalled in post-build actions it may be that the node momentarily lost connection with the master. We have seen this with our Linux nodes running Open JDK. The problem was resolved by switching to Oracle's JDK. To do this, take the node temporarily offline and SSH into the box. Then run the following commands to remove Open JDK and install Oracle's JDK:
 
    sudo apt-get purge openjdk*
    sudo add-apt-repository ppa:webupd8team/java
    sudo apt-get update
    sudo apt-get install oracle-java7-installer
 
You can confirm the version of Java using the command:
 
    java -version
 
Once this is done, you can launch the node again from Jenkins web console.


= References =
== PIP installation is slow ==
Some content has been taken from [https://wiki.jenkins-ci.org/display/JENKINS/Thanks+for+using+OSX+Installer here].
If Python packages are slow to install you should check that PIP mirrors are not enabled. If not, it may be that the main PyPI mirror is slow to respond.

Latest revision as of 13:22, 6 October 2016

Summary

Jenkins CI is used by Web QA to run automated tests. The production instance is publicly accessible and can be found here. Details of the setup and administration are available on our internal Mana page. We also have a staging instance for adhoc jobs and experimentation. This is secured behind Mozilla VPN, and can be found here. Documentation for the staging instance is also available on Mana.

Plugins

Below is a list of plugins in use (dependencies have been omitted):

Global Environment Variables

Several environment variables are configured:

  • APPIUM_BROWSER_NAME - Browser name to use with Appium. The default is Browser.
  • APPIUM_DEVICE_NAME - Device name to use with Appium. The default is Android Emulator.
  • APPIUM_PLATFORM_NAME - Platform name to use with Appium. The default is Android.
  • APPIUM_PLATFORM_VERSION - Platform version to use with Appium. The default is 5.1.
  • APPIUM_VERSION - Version of Appium to use. The default is 1.5.3.
  • FIREFOX_VERSION - Firefox version to use. The default is 47.0.
  • MAC_VERSION - MacOS version to use. The default is OS X 10.11.
  • PIP_INDEX_URL - Sets the PyPI index URL. The default is empty, to use the main PyPI index.
  • PIP_USE_MIRRORS - Controls use of the PyPI mirrors. By default this is false to speed up PIP installations.
  • PIP_VERBOSE - Controls the verbosity of PIP output. By default this is false and useful to enable for debugging.
  • PYTEST_ADDOPTS - Add py.test options to all jobs. The default is -r=fsxXR --verbose --tb=short.
  • PYTEST_PROCESSES - Number of parallel py.test processes to use. The default is 15.
  • PYTEST_RERUNS - Number of times to rerun a failed test (used by pytest-rerunfailures). The default is 5.
  • SAUCELABS_USERNAME - Username to use for Sauce Labs.
  • SELENIUM_EXCLUDE_DEBUG - Exclude capturing these debug types from Selenium tests. The default is logs.
  • SELENIUM_VERSION - Selenium version to use. The default is 2.53.1.
  • TOX_TESTENV_PASSENV - Environment variables to pass down into Tox environments. The default is http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY SAUCELABS_USERNAME SAUCELABS_API_KEY.
  • WINDOWS_VERSION - Windows version to use. The default is Windows 7.

Known Issues

  • None

Troubleshooting

Try enabling verbose PIP output to determine if installation of Python packages is causing issues.

PIP index is unavailable

We use the main PyPI (Python Package Index) for installing Python packages. If this experiences issues then we can enable use of PIP mirrors.

PIP installation is slow

If Python packages are slow to install you should check that PIP mirrors are not enabled. If not, it may be that the main PyPI mirror is slow to respond.

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Execution/Web_Testing/Automation/Jenkins&oldid=1150443"