Auto-tools/Projects/W3C CSS Test Mirroring: Difference between revisions
| Line 11: | Line 11: | ||
** Because these newly downloaded tests may exist in our upstream directory we need to ensure that the reftest manifest annotations those files had in the upstream directory are added to the generated manifest for the new file in the tests/ directory. | ** Because these newly downloaded tests may exist in our upstream directory we need to ensure that the reftest manifest annotations those files had in the upstream directory are added to the generated manifest for the new file in the tests/ directory. | ||
** Once the manifests are merged, the upstream test files (and their corresponding manifest entiries) should be deleted. | ** Once the manifests are merged, the upstream test files (and their corresponding manifest entiries) should be deleted. | ||
== Other considerations == | |||
We probably want to sync the W3C tests back into some kind of project branch. This way we can run the automated tests on them and let developers work on them to get them running and passing. The tests may need some specific Gecko reftest manifest annotation or something to get our tests to green. Once we get the tests to green, we can then merge them into mozilla-central. This last merge would be a manual step. | |||
= Goal = | = Goal = | ||
Revision as of 23:41, 16 September 2011
Rationale
The W3C CSS Test working group has a bunch of tests in their hg repo on the W3C site. We need an automated way to pull the tests back from the W3C site into our infrastructure so that these tests can be run in addition to the rest of our automated test suites.
We also need a way to automatically annotate new tests for inclusion into the upstream W3C repo. As a point of process, when a test is submitted to the W3C, it goes into a "submitted" tree, and then once it is accepted it is in an "accepted" tree. This is important to remember for the process outlined below.
Proposed Process
- Developer writes test for W3C - these have a separate license, so they land in the other-licenses/ directory (Elika will determine the proper directory for this), for now, let's call it other-licenses/w3c-css/upstream
- An hg hook watches for checkins to this directory and causes the new test to be pushed into the W3C "submitted" tree.
- On some time sequence (once a day or something) we pull from the W3C "accepted" tree into our other-licenses/w3c-css/tests (name TBD) directory.
- When we pull the tests down, we need to generate the reftest manifests for the newly downloaded tests.
- Because these newly downloaded tests may exist in our upstream directory we need to ensure that the reftest manifest annotations those files had in the upstream directory are added to the generated manifest for the new file in the tests/ directory.
- Once the manifests are merged, the upstream test files (and their corresponding manifest entiries) should be deleted.
Other considerations
We probably want to sync the W3C tests back into some kind of project branch. This way we can run the automated tests on them and let developers work on them to get them running and passing. The tests may need some specific Gecko reftest manifest annotation or something to get our tests to green. Once we get the tests to green, we can then merge them into mozilla-central. This last merge would be a manual step.
Goal
- Have an automated way to keep our W3C CSS reftests in sync with what is up in the "accepted" directory of the W3C and also help us have a mechanism to upload tests automatically.
Non-Goal
- Not to solve every corner case. If the W3C ends up changing the name of the test between the "submitted" and the "accepted" step, we may lose the ability to track the test, and we'll have to deal with that manually.
More Info
- Talk to Elika about this - fantasai online