Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(3535)

Unified Diff: docs/testing/web_platform_tests.md

Issue 2695963002: Update web-platform-tests docs (Closed)
Patch Set: Created 3 years, 10 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View side-by-side diff with in-line comments
Download patch
« no previous file with comments | « docs/testing/test_browser_dialog.md ('k') | docs/testing/writing_layout_tests.md » ('j') | no next file with comments »
Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
Index: docs/testing/web_platform_tests.md
diff --git a/docs/testing/web_platform_tests.md b/docs/testing/web_platform_tests.md
index 878ef50fcdd4ac7f03508760499f372db4ecf4cd..e20e83af9f3288037064820a1436630ec1161a4a 100644
--- a/docs/testing/web_platform_tests.md
+++ b/docs/testing/web_platform_tests.md
@@ -11,9 +11,14 @@ test suite where all browser engines are collaborating. There's also a
that will [soon be merged into
web-platform-tests](https://github.com/w3c/csswg-test/issues/1102).
+Chromium has 2-way import/export process with the upstream web-platform-tests
+repository, where tests are imported into
+[LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
+and any changes to the imported tests are also exported to web-platform-tests.
+
[TOC]
-## Import
+## Importing tests
Chromium has mirrors
([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platform-tests/),
@@ -21,7 +26,7 @@ Chromium has mirrors
GitHub repos, and periodically imports a subset of the tests so that they are
run as part of the regular Blink layout test testing process.
-The goal of this process are to be able to run the Web Platform Tests unmodified
+The goal of this process are to be able to run Web Platform Tests unmodified
locally just as easily as we can run the Blink tests, and ensure that we are
tracking tip-of-tree in the Web Platform Tests repository as closely as
possible, and running as many of the tests as possible.
@@ -39,6 +44,8 @@ The easiest way to check the status of recent imports is to look at:
- Recent CLs created by
[blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/search?owner=blink-w3c-test-autoroller%40chromium.org).
+Automatic imports are intended to run at least once every 24 hours.
+
### Skipped tests
We control which tests are imported via a file called
@@ -70,17 +77,17 @@ it, and on the next auto-import, the new tests should be imported.
If you want to import immediately (in order to try the tests out locally, etc)
you can also run `wpt-import --allow-local-commits`, but this is not required.
-## Contributing tests back to the Web Platform Tests project.
+## Writing tests
-If you need to make changes to Web Platform Tests, just commit your changes
-directly to
-[LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
+To contribute changes to Web Platform Tests, just commit your changes directly
+to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
and the changes will be automatically upstreamed within 24 hours.
-Note that tests in Web Platform Tests are expected to match behavior defined by
-the relevant WHATWG or W3C specification, not simply Blink's behavior. If in
-doubt, please request code review from someone with expertise in the relevant
-specification text.
+Changes involving adding, removing or modifying tests can all be upstreamed.
+Any changes outside of
+[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be
+upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
+will also not be upstreamed.
Note: if you're adding a new test in `external/wpt`, you'll need to re-generate
MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is
@@ -91,18 +98,47 @@ Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \
--tests-root=LayoutTests/external/wpt
```
-### What kinds of changes can be upstreamed?
+Most tests are written using testharness.js, see
+[Writing Layout Tests](./writing_layout_tests.md) and
+[Layout Tests Tips](./layout_tests_tips.md) for general guidelines.
-In general, changes involving adding, removing or modifying tests can all be
-upstreamed. From a Chromium commit, any changes outside of
-[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be
-upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
-will also not be upstreamed.
+### Write tests against specifications
+
+Tests in Web Platform Tests are expected to match behavior defined by the
+relevant specification. In other words, all assertions that a test makes
+should be derived from a specification's normative requirements, and not go
+beyond them. It is often necessary to change the specification to clarify what
+is and isn't required.
+
+When the standards discussion is still ongoing or blocked on some implementation
+successfully shipping the hoped-for behavior, write the tests outside of
+web-platform-tests and upstream them when the specification is finally updated.
+Optionally, it may be possible to write deliberately failing tests against the
+current specification and later update them.
+
+### Tests that require testing APIs
+
+Tests that depend on `internals.*`, `eventSender.*` or other internal testing
+APIs cannot yet be written as part of web-platform-tests.
+
+An alternative is to write manual tests that are automated with scripts from
+[wpt_automation](../../third_party/WebKit/LayoutTests/external/wpt_automation).
+Such tests still require case-by-case automation to run for other browser
+engines, but are more valuable than purely manual tests.
+
+*** note
+TODO(foolip): Figure out and document a more scalable test automation solution.
+***
+
+### Adding new top-level directories
Entirely new top-level directories should generally be added upstream, since
that's the only way to add an OWNERS file upstream. After adding a new top-level
directory upstream, you should add a line for it in `W3CImportExpectations`.
+Adding the new directory (and `W3CImportExpectations` entry) in Chromium and
+later adding an OWNERS file upstream also works.
+
### Will the exported commits be linked to my GitHub profile?
The email you commit with in Chromium will be the author of the commit on
@@ -123,3 +159,18 @@ It's still possible to make direct pull requests to web-platform-tests. The
processes for getting new tests committed the W3C repos are at
http://testthewebforward.org/docs/. Some specifics are at
http://testthewebforward.org/docs/github-101.html.
+
+## Reviewing tests
+
+Anyone who can review code and tests in Chromium can also review changes in
+[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
+that will be automatically upstreamed. There will be no additional review in
+web-platform-tests as part of the export process.
+
+If upstream reviewers have feedback on the changes, discuss on the pull request
+created during export, and if necessary work on a new pull request to iterate
+until everyone is satisfied.
+
+When reviewing tests, check that they match the relevant specification, which
+may not fully match the implementation. See also
+[Write tests against specifications](#Write-tests-against-specifications).
« no previous file with comments | « docs/testing/test_browser_dialog.md ('k') | docs/testing/writing_layout_tests.md » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698