Auto-tools/Projects/Autolog: Difference between revisions

Jump to navigation Jump to search

Auto-tools/Projects/Autolog (view source)

Revision as of 19:19, 17 May 2011

2,547 bytes removed ,  17 May 2011
→‎Open Issues
No edit summary
Line 121: Line 121:


=== Open Issues ===
=== Open Issues ===
==== Identifying 'testruns' ====
The above structure would work fine for displaying a TBPL-like result view.  It might be problematic if we intend to feed into OrangeFactor, however.
The problem is in identifying unique test runs.  For OrangeFactor, we rely on the fact the buildbot uses the same buildid with all related 'testgroups'.  If buildbot reruns the same testgroups (because it's bored), it generates a new buildid, even though the revision is still the same.  Thus we can identify unique test runs.
For non-buildbot cases (and I'm specifically thinking of Crossweave), we don't have an analogous buildid.  In the Crossweave case, each 'testgroup' fired off against a given revision is independent and doesn't share any metadata (like a buildid) with other testgroups run at the same time.
If we want to maintain consistency with OrangeFactor, we may have to require a 'buildid' value, which would be the same across all testgroups which are initiated by the same event.  This would require some refactoring of Crossweave and possibly other tools.
Using 'buildid' isn't a perfect solution, though, as some buildbot jobs (like once-a-day win64 runs) use a distinct buildid that doesn't match other buildbot jobs run on the same changeset.  We're currently excluding these in OrangeFactor so it isn't skewing data, but it illustrates the drawbacks of relying on buildid.
The other option is to change the way OrangeFactor identifies testruns. Instead of relying on buildid, we could implement some algorithm like this:
# identify all the changesets that have testgroups; the number of changesets is our preliminary testrun_count
# for each changeset, identify the list of testgroups (L) and the unique set of testgroups (S), based on 'testgroup' and 'platform'
# if len(S) > len(L), create a list of 'extra testgroups' (E) by removing members of (S) from (L)
# sort (E) and look for the maximum number of duplicate entries (e.g., if (E) contains three 'mochitest-other'/'win32' and two 'mochitest-1/5'/'linux64', return 3), and add this number of the preliminary testrun_count
There's always going to be some guesswork involved, since the idea of a 'testrun' we're using in OrangeFactor is entirely an intellectual concept and not something supported intrinsically by our test frameworks.
'''update''': Per the WOO meeting on 2011-03-02, we decided to add a 'testrun' field to the data structure above, which the implementation would be responsible for filling with a value that is unique for specific collections of testgroups.


==== Log Storage ====
==== Log Storage ====
Jgriffin
Confirmed users
1,905

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/309695"

Navigation menu