EngineeringProductivity/Projects/Treeherder: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (→‎Possible Future Milestones: - added tier 2 support to the future list.)
(Clean up out of date sections)
 
(48 intermediate revisions by 8 users not shown)
Line 1: Line 1:
== Overview ==
== About ==
[https://treeherder.mozilla.org/ Treeherder] is the successor to [[Sheriffing/TBPL|TBPL]].
[https://treeherder.mozilla.org/ Treeherder] is a reporting dashboard for checkins to Mozilla projects (for example, [https://developer.mozilla.org/en-US/docs/mozilla-central mozilla-central]). It allows users to see the results of automatic builds and their respective tests. Treeherder also provides a rich set of APIs that can be used by other projects interested in this information.


The goal of the first version of TBPL was to correlate checkins with Buildbot results, so that developers and sheriffs could quickly see test results across multiple platforms for automatic builds associated with a developer's specific Firefox changesets. As more test types, platforms, products and features were added to TBPL, a variety of architectural bottle necks became apparent.  
Treeherder is the successor to [[Sheriffing/TBPL|TBPL]].


The purpose of Treeherder is to support longer term use-cases for managing and visualizing data and rewrite the existing application from scratch, so that it can scale appropriately in the future.
For tracking performance data, see Treeherder's sister project, [[EngineeringProductivity/Projects/Perfherder|Perfherder]].
 
For tracking intermittent test failures, see [[EngineeringProductivity/Projects/IntermittentFailuresView|Intermittent Failures View]].


== Contributing ==
== Contributing ==
* Good first bugs for new developers: [http://www.joshmatthews.net/bugsahoy/?reporting=1 Bugs Ahoy]
To make UI-only changes, you only need to perform a very simple setup running a local webserver pointing at the production backend, described [https://treeherder.readthedocs.io/installation.html#ui-development here]. If you wish to hack on the backend, or the UI and backend together, you will instead need to set up a Vagrant environment, using [https://treeherder.readthedocs.io/installation.html#server-and-full-stack-development these steps].
* Bugzilla: [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder File a bug] / [https://bugzilla.mozilla.org/buglist.cgi?product=Tree+Management&f1=component&v1=Treeherder&o1=substring&resolution=--- All open bugs] / [https://bugzilla.mozilla.org/buglist.cgi?order=Bug%20Number&resolution=FIXED&chfieldto=Now&chfield=resolution&chfieldfrom=-14d&chfieldvalue=FIXED&f1=component&v1=Treeherder&o1=substring Recently fixed] / [[#Bug_Tables|Bug tables]]
 
* IRC: [irc://irc.mozilla.org/treeherder #treeherder] / [http://logs.glob.uno/?c=treeherder logs]
* [https://ateam-bootcamp.readthedocs.io/en/latest/ A-Team Bootcamp]: Best practices for working on A-Team projects (of which Treeherder is one), including valuable information on using Git and Bugzilla. If you're new to Mozilla or the A-Team, please read this guide thoroughly before proceeding.
* Meetings: [[Auto-tools/Projects/Treeherder/Meetings|Notes & dial-in details]]
* Good first bugs for new developers: [https://codetribute.mozilla.org/projects/treeherder Codetribute]
* Mozilla instances: [http://treeherder-dev.allizom.org Dev] / [https://treeherder.allizom.org Staging] / [https://treeherder.mozilla.org Production]
* Issue tracker: [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder Report a bug] / [https://bugzilla.mozilla.org/query.cgi?query_format=advanced&product=Tree+Management&f1=component&o1=substring&v1=Treeherder&resolution=--- search open bugs]
* Mozilla Treeherder instances: [https://treeherder.allizom.org Staging] / [https://treeherder.mozilla.org Production] / [https://whatsdeployed.io/s-dqv Deployed changes info]


== Source and Docs ==
== Source and Docs ==
* Treeherder backend: [https://github.com/mozilla/treeherder-service Source] / [https://treeherder-service.readthedocs.org Docs] / [http://treeherder-dev.allizom.org/docs/ API Docs] / [https://mana.mozilla.org/wiki/display/websites/treeherder.mozilla.org Production mana page]
* UI & backend: [https://github.com/mozilla/treeherder Source] / [https://treeherder.readthedocs.io Docs]
* Treeherder UI: [https://github.com/mozilla/treeherder-ui Source] / [https://treeherder-ui.readthedocs.org Docs]
* Treeherder data submission clients:
* Treeherder data submission clients:
** Python: [https://github.com/mozilla/treeherder-client Source] / [https://github.com/mozilla/treeherder-client/blob/master/README.md Readme]
** Pulse: [https://hg.mozilla.org/automation/mozillapulse/ Source] / [http://treeherder.readthedocs.io/submitting_data.html#submitting-using-pulse docs]
** NodeJS: [https://github.com/mozilla/treeherder-node Source] / [https://github.com/mozilla/treeherder-node/blob/master/README.md Readme]
** (Python and Node submission support has been removed.)
 
== Roadmap ==
==== Milestone M1: Functional Prototype ====
* Deliver a functional prototype that sheriffs can use for E2E testing.
* Status: {{done|}} delivered 2014 Q1.
==== Milestone M2: Production Deployment ====
* Complete a security review and deploy into production with IT.
* Status: {{done|}} delivered 2014 Q2.
==== Milestone M3: Sheriff Transition from TBPL ====
* Work with the sheriffs to address bugs and workflow issues so that Treeherder is a viable alternate for TBPL for their workflow, and transition the sheriffs to using Treeherder for their day-to-day work. Bugs: see dependents of {{bug|1030636}}.
* Status: {{done|}} delivered 2014 Q3 ([https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/WSQY2EYgxz8 newsgroup post]).
 
==== Milestone M4: Make TBPL redundant for all users ====
* Primary goal:
** M4.1: Work to implement other developer use cases, refine sheriffing work flows, and make TBPL redundant for all of its users. (Note: [[Sheriffing/TBPL#TBPL_switch-off_tasks|Switching off TBPL]] will come later). Bugs: see [[#Developer_transition_blockers]] below.
* Secondary goals (may be deferred to later milestones):
** M4.2: Improve support for submitting correctness data from non-buildbot sources such as TaskCluster and Autophone. Bugs: see [[#Data_submission_bugs]] below.
** M4.3: General cleanup to make Treeherder development more accessible to others. Bugs: see [[#Hacking_UX_bugs]] below.
* Status: {{ok|}} for delivery 2015 Q1.


==== Milestone M5: Performance Data ====
== Getting in touch ==
* We want to obsolete Graphserver and Datazilla, and make Treeherder the dashboard that is used to visualize performance data generated by Talos and other sources. This will be comprised of at least the following:
* Chat on IRC: [irc://irc.mozilla.org/treeherder #treeherder] / [[IRC|learn about IRC]] / [https://mozilla.logbot.info/treeherder channel logs]
** A performance panel in pushlog view that can be used to see details about the performance data submitted for a particular push, plus recent antecedents.
* Mailing list: [https://lists.mozilla.org/listinfo/tools-treeherder tools-treeherder] (or [https://groups.google.com/forum/#!forum/mozilla.tools.treeherder via Google groups])
** A performance summary tab that gives a high-level overview by branch, platform, and suite.
* Weekly meetings: [[EngineeringProductivity/Projects/Treeherder/Meetings|Notes & dial-in details]]
* Status: {{proposed|}} detailed planning to take place in late 2014 Q4, with delivery in 2015 Q1.
==== Possible Future Milestones ====
* Annotation of backouts, tree closures
* Better performance graphs
* Integration of performance alerts
* Integration of OrangeFactor, GoFaster dashboards
* Create a display for code coverage data
* Additional metrics and visualizations
* Tier 2 jobs support


== Bug Tables ==
== What we're working on ==
We're still refining our use of the priority flag, but this is the rough rule of thumb used when triaging:
Assigned Treeherder bugs that have been modified in the last month.
* '''P1:''' Bugs that should take absolute priority and mean other work is put on hold. Ideal time frame for work beginning: This week.
[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Atreeherder+-assignee%3Anobody%40mozilla.org+delta_ts%3E-1m View on Bugzilla]
* '''P2:''' Important "next" bugs. Time-frame: Within a month.
<bugzilla stats="hide">
* '''P3:''' Time frame: Ideally in the next few months.
* '''P4:''' Time frame: Next quarter or two.
* '''P5:'''  Time frame: Unknown / future.
Once one queue (eg P1) is nearing depletion, a selection of the bugs from the group below will be promoted to that priority level. Once the substantial lists of [[#Developer_transition_blockers]] and infrastructure bugs have been reduced, the guide above may change.
 
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Atreeherder+-assignee%3Anobody%40mozilla.org Assigned] ===
A summary of Treeherder bugs that are being worked on at the moment.
<bugzilla>
{
"component": "Treeherder", "component_type": "contains",
"resolution": "---",
"email1": "nobody@mozilla.org", "email1_type": "not_equals", "email1_assigned_to": "1",
"include_fields": "id,priority,component,summary,assigned_to",
"order": "priority,assigned_to"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Atreeherder+P1+assignee%3Anobody%40mozilla.org Unassigned P1] ===
All open P1 Treeherder bugs that do not have an assignee. Ideally these are the bugs that should be worked on next, once the currently assigned bugs are fixed.
<bugzilla>
{
"component": "Treeherder", "component_type": "contains",
"resolution": "---",
"email1": "nobody@mozilla.org", "email1_type": "equals", "email1_assigned_to": "1",
"priority": "P1",
"include_fields": "id,component,summary",
"order": "component"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Atreeherder+priority%3A--+assignee%3Anobody%40mozilla.org Needing Triage] ===
All bugs that do not have a priority set and are unassigned (excluding Perfherder). These are ideally triaged weekly.
<bugzilla>
{
"quicksearch": ":treeherder -:perfherder priority:-- assignee:nobody@mozilla.org",
"include_fields": "id,component,summary",
"order": "component"
}
</bugzilla>
 
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1059400 Developer transition blockers] ===
Treeherder bugs blocking remaining users of TBPL agreeing to make the transition to Treeherder, as well as fixing use-cases for which regular Treeherder users still have to fall back to TBPL from time to time.
<bugzilla>
{
"resolution": "---",
"blocked": "1059400",
"include_fields": "id,priority,component,summary,assigned_to",
"order": "assigned_to,priority"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Atreeherder All open bugs] ===
<bugzilla>
{
"component": "Treeherder", "component_type": "contains",
"resolution": "---",
"include_fields": "id,priority,component,summary,assigned_to"
}
</bugzilla>
 
=== [https://bugzilla.mozilla.org/buglist.cgi?order=Bug%20Number&resolution=FIXED&chfieldto=Now&chfield=resolution&chfieldfrom=-14d&chfieldvalue=FIXED&f1=component&v1=Treeherder&o1=substring Recently fixed] ===
Bugs changed to "Resolved FIXED" in the last 14 days.
<bugzilla>
{
{
   "component": "Treeherder", "component_type": "contains",
   "component": "Treeherder", "component_type": "contains",
   "resolution": "FIXED",
   "resolution": "---",
   "changed_after": "-14d",
  "email1": "nobody@mozilla.org", "email1_type": "not_equals", "email1_assigned_to": "1",
   "changed_after": "-1m",
   "changed_before": "Now",
   "changed_before": "Now",
  "changed_field": "resolution",
   "include_fields": "id,priority,component,summary,assigned_to",
  "changed_field_to": "FIXED",
   "order": "priority,assigned_to"
   "include_fields": "id,component,summary,assigned_to",
   "order": "assigned_to,id"
}
}
</bugzilla>
</bugzilla>
== Archive ==
* Original design notes:
** [[Auto-tools/Projects/Treeherder/Current_TBPL_Feature_List]]
** [[Auto-tools/Projects/Treeherder/Design_Draft]] (quite out of date now)
* [https://www.pivotaltracker.com/projects/749519 Pivotal Tracker project] (Deprecated)
* Etherpads: [https://etherpad.mozilla.org/sheriffing-with-treeherder sheriff-feedback-april] / [https://sheriffs.etherpad.mozilla.org/move-to-treeherder-requirements sheriff-feedback-june] / [https://etherpad.mozilla.org/treeherder-transition sheriff-transition-meeting-notes]
* Treeherder debrief Nov 2014 [https://etherpad.mozilla.org/treeherder-debrief treeherder-debrief]

Latest revision as of 06:42, 11 March 2019

About

Treeherder is a reporting dashboard for checkins to Mozilla projects (for example, mozilla-central). It allows users to see the results of automatic builds and their respective tests. Treeherder also provides a rich set of APIs that can be used by other projects interested in this information.

Treeherder is the successor to TBPL.

For tracking performance data, see Treeherder's sister project, Perfherder.

For tracking intermittent test failures, see Intermittent Failures View.

Contributing

To make UI-only changes, you only need to perform a very simple setup running a local webserver pointing at the production backend, described here. If you wish to hack on the backend, or the UI and backend together, you will instead need to set up a Vagrant environment, using these steps.

Source and Docs

  • UI & backend: Source / Docs
  • Treeherder data submission clients:
    • Pulse: Source / docs
    • (Python and Node submission support has been removed.)

Getting in touch

What we're working on

Assigned Treeherder bugs that have been modified in the last month. View on Bugzilla

No results.

Retrieved from "https://wiki.allizom.org/index.php?title=EngineeringProductivity/Projects/Treeherder&oldid=1208668"