Auto-tools/Projects/W3C CSS Test Mirroring: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(2 intermediate revisions by 2 users not shown)
Line 2: Line 2:
The W3C CSS Test working group has a bunch of tests in their hg repo on the W3C site.  We need an automated way to pull the tests back from the W3C site into our infrastructure so that these tests can be run in addition to the rest of our automated test suites.
The W3C CSS Test working group has a bunch of tests in their hg repo on the W3C site.  We need an automated way to pull the tests back from the W3C site into our infrastructure so that these tests can be run in addition to the rest of our automated test suites.


We also need a way to automatically annotate new tests for inclusion into the upstream W3C repo.  As a point of process, when a test is submitted to the W3C, it goes into a "submitted" tree, and then once it is accepted it is in an "accepted" tree.  This is important to remember for the process outlined below.
We also need a way to automatically annotate new tests for inclusion into the upstream W3C repo.  As a point of process, when a test is submitted to the W3C, it goes into a "submitted" tree, and then once it is accepted it is in an "approved" tree.  This is important to remember for the process outlined below.


= Proposed Process =
= Proposed Process =
* Developer writes test for W3C - these have a separate license, so they land in the other-licenses/ directory (Elika will determine the proper directory for this), for now, let's call it other-licenses/w3c-css/upstream
* Developer writes test to be synced to W3C - these have a separate license, so they land in the other-licenses/ directory (fantasai will determine the proper directory for this), for now, let's call it other-licenses/w3c-css/submitted
* An hg hook watches for checkins to this directory and causes the new test to be pushed into the W3C "submitted" tree.
* An hg hook watches for checkins to this directory and causes the new test to be pushed into the W3C "submitted" tree.
* On some time sequence (once a day or something) we pull from the W3C "accepted" tree into our other-licenses/w3c-css/tests (name TBD) directory.
* On some time sequence (once a day or something) we pull from the W3C "approved" tree into our other-licenses/w3c-css/received (name TBD) directory.
** When we pull the tests down, we need to generate the reftest manifests for the newly downloaded tests.
** When we pull the tests down, we need to generate the reftest manifests for the newly downloaded tests. (Note: W3C has build scripts that can generate reftest manifests.)
** Because these newly downloaded tests may exist in our upstream directory we need to ensure that the reftest manifest annotations those files had in the upstream directory are added to the generated manifest for the new file in the tests/ directory.
** Because these newly downloaded tests may exist in our submitted directory we need to ensure that the reftest manifest annotations those files had in the submitted directory are added to the generated manifest for the new file in the received/ directory.
** Once the manifests are merged, the upstream test files (and their corresponding manifest entiries) should be deleted.
** Once the manifests are merged, the submitted test files (and their corresponding manifest entries) should be deleted.
 
== Other considerations ==
We probably want to sync the W3C tests back into some kind of project branch.  This way we can run the automated tests on them and let developers work on them to get them running and passing.  The tests may need some specific Gecko reftest manifest annotation or something to get our tests to green.  Once we get the tests to green, we can then merge them into mozilla-central.  This last merge would be a manual step.


= Goal =
= Goal =
* Have an automated way to keep our W3C CSS reftests in sync with what is up in the "accepted" directory of the W3C and also help us have a mechanism to upload tests automatically.
* Incorporate W3C-hosted CSS reftests into our automated testing framework
* Have a place and process for devs to land tests that should be submitted to W3C
* Keep our copy and W3C's copy of the tests in sync


= Non-Goal =
= Non-Goal =
Line 19: Line 24:


= More Info =
= More Info =
* Talk to Elika about this - fantasai online
* CSSWG test server - http://test.csswg.org/
* CSSWG test wiki - http://wiki.csswg.org/test
* Bug for legal signoff - https://bugzilla.mozilla.org/show_bug.cgi?id=691950
* Questions? Ask [http://fantasai.inkedblade.net/contact fantasai]

Latest revision as of 01:21, 30 October 2011

Rationale

The W3C CSS Test working group has a bunch of tests in their hg repo on the W3C site. We need an automated way to pull the tests back from the W3C site into our infrastructure so that these tests can be run in addition to the rest of our automated test suites.

We also need a way to automatically annotate new tests for inclusion into the upstream W3C repo. As a point of process, when a test is submitted to the W3C, it goes into a "submitted" tree, and then once it is accepted it is in an "approved" tree. This is important to remember for the process outlined below.

Proposed Process

  • Developer writes test to be synced to W3C - these have a separate license, so they land in the other-licenses/ directory (fantasai will determine the proper directory for this), for now, let's call it other-licenses/w3c-css/submitted
  • An hg hook watches for checkins to this directory and causes the new test to be pushed into the W3C "submitted" tree.
  • On some time sequence (once a day or something) we pull from the W3C "approved" tree into our other-licenses/w3c-css/received (name TBD) directory.
    • When we pull the tests down, we need to generate the reftest manifests for the newly downloaded tests. (Note: W3C has build scripts that can generate reftest manifests.)
    • Because these newly downloaded tests may exist in our submitted directory we need to ensure that the reftest manifest annotations those files had in the submitted directory are added to the generated manifest for the new file in the received/ directory.
    • Once the manifests are merged, the submitted test files (and their corresponding manifest entries) should be deleted.

Other considerations

We probably want to sync the W3C tests back into some kind of project branch. This way we can run the automated tests on them and let developers work on them to get them running and passing. The tests may need some specific Gecko reftest manifest annotation or something to get our tests to green. Once we get the tests to green, we can then merge them into mozilla-central. This last merge would be a manual step.

Goal

  • Incorporate W3C-hosted CSS reftests into our automated testing framework
  • Have a place and process for devs to land tests that should be submitted to W3C
  • Keep our copy and W3C's copy of the tests in sync

Non-Goal

  • Not to solve every corner case. If the W3C ends up changing the name of the test between the "submitted" and the "accepted" step, we may lose the ability to track the test, and we'll have to deal with that manually.

More Info

Retrieved from "https://wiki.allizom.org/index.php?title=Auto-tools/Projects/W3C_CSS_Test_Mirroring&oldid=362276"