Index: docs/testing/web_platform_tests.md |
diff --git a/docs/testing/web_platform_tests.md b/docs/testing/web_platform_tests.md |
index 878ef50fcdd4ac7f03508760499f372db4ecf4cd..e20e83af9f3288037064820a1436630ec1161a4a 100644 |
--- a/docs/testing/web_platform_tests.md |
+++ b/docs/testing/web_platform_tests.md |
@@ -11,9 +11,14 @@ test suite where all browser engines are collaborating. There's also a |
that will [soon be merged into |
web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). |
+Chromium has 2-way import/export process with the upstream web-platform-tests |
+repository, where tests are imported into |
+[LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
+and any changes to the imported tests are also exported to web-platform-tests. |
+ |
[TOC] |
-## Import |
+## Importing tests |
Chromium has mirrors |
([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platform-tests/), |
@@ -21,7 +26,7 @@ Chromium has mirrors |
GitHub repos, and periodically imports a subset of the tests so that they are |
run as part of the regular Blink layout test testing process. |
-The goal of this process are to be able to run the Web Platform Tests unmodified |
+The goal of this process are to be able to run Web Platform Tests unmodified |
locally just as easily as we can run the Blink tests, and ensure that we are |
tracking tip-of-tree in the Web Platform Tests repository as closely as |
possible, and running as many of the tests as possible. |
@@ -39,6 +44,8 @@ The easiest way to check the status of recent imports is to look at: |
- Recent CLs created by |
[blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/search?owner=blink-w3c-test-autoroller%40chromium.org). |
+Automatic imports are intended to run at least once every 24 hours. |
+ |
### Skipped tests |
We control which tests are imported via a file called |
@@ -70,17 +77,17 @@ it, and on the next auto-import, the new tests should be imported. |
If you want to import immediately (in order to try the tests out locally, etc) |
you can also run `wpt-import --allow-local-commits`, but this is not required. |
-## Contributing tests back to the Web Platform Tests project. |
+## Writing tests |
-If you need to make changes to Web Platform Tests, just commit your changes |
-directly to |
-[LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
+To contribute changes to Web Platform Tests, just commit your changes directly |
+to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
and the changes will be automatically upstreamed within 24 hours. |
-Note that tests in Web Platform Tests are expected to match behavior defined by |
-the relevant WHATWG or W3C specification, not simply Blink's behavior. If in |
-doubt, please request code review from someone with expertise in the relevant |
-specification text. |
+Changes involving adding, removing or modifying tests can all be upstreamed. |
+Any changes outside of |
+[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be |
+upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, |
+will also not be upstreamed. |
Note: if you're adding a new test in `external/wpt`, you'll need to re-generate |
MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is |
@@ -91,18 +98,47 @@ Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ |
--tests-root=LayoutTests/external/wpt |
``` |
-### What kinds of changes can be upstreamed? |
+Most tests are written using testharness.js, see |
+[Writing Layout Tests](./writing_layout_tests.md) and |
+[Layout Tests Tips](./layout_tests_tips.md) for general guidelines. |
-In general, changes involving adding, removing or modifying tests can all be |
-upstreamed. From a Chromium commit, any changes outside of |
-[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be |
-upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, |
-will also not be upstreamed. |
+### Write tests against specifications |
+ |
+Tests in Web Platform Tests are expected to match behavior defined by the |
+relevant specification. In other words, all assertions that a test makes |
+should be derived from a specification's normative requirements, and not go |
+beyond them. It is often necessary to change the specification to clarify what |
+is and isn't required. |
+ |
+When the standards discussion is still ongoing or blocked on some implementation |
+successfully shipping the hoped-for behavior, write the tests outside of |
+web-platform-tests and upstream them when the specification is finally updated. |
+Optionally, it may be possible to write deliberately failing tests against the |
+current specification and later update them. |
+ |
+### Tests that require testing APIs |
+ |
+Tests that depend on `internals.*`, `eventSender.*` or other internal testing |
+APIs cannot yet be written as part of web-platform-tests. |
+ |
+An alternative is to write manual tests that are automated with scripts from |
+[wpt_automation](../../third_party/WebKit/LayoutTests/external/wpt_automation). |
+Such tests still require case-by-case automation to run for other browser |
+engines, but are more valuable than purely manual tests. |
+ |
+*** note |
+TODO(foolip): Figure out and document a more scalable test automation solution. |
+*** |
+ |
+### Adding new top-level directories |
Entirely new top-level directories should generally be added upstream, since |
that's the only way to add an OWNERS file upstream. After adding a new top-level |
directory upstream, you should add a line for it in `W3CImportExpectations`. |
+Adding the new directory (and `W3CImportExpectations` entry) in Chromium and |
+later adding an OWNERS file upstream also works. |
+ |
### Will the exported commits be linked to my GitHub profile? |
The email you commit with in Chromium will be the author of the commit on |
@@ -123,3 +159,18 @@ It's still possible to make direct pull requests to web-platform-tests. The |
processes for getting new tests committed the W3C repos are at |
http://testthewebforward.org/docs/. Some specifics are at |
http://testthewebforward.org/docs/github-101.html. |
+ |
+## Reviewing tests |
+ |
+Anyone who can review code and tests in Chromium can also review changes in |
+[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
+that will be automatically upstreamed. There will be no additional review in |
+web-platform-tests as part of the export process. |
+ |
+If upstream reviewers have feedback on the changes, discuss on the pull request |
+created during export, and if necessary work on a new pull request to iterate |
+until everyone is satisfied. |
+ |
+When reviewing tests, check that they match the relevant specification, which |
+may not fully match the implementation. See also |
+[Write tests against specifications](#Write-tests-against-specifications). |