| OLD | NEW |
| 1 # Web Platform Tests | 1 # web-platform-tests |
| 2 | 2 |
| 3 Interoperability between browsers is | 3 Interoperability between browsers is |
| 4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's | 4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's |
| 5 mission of improving the web. We believe that leveraging and contributing to a | 5 mission of improving the web. We believe that leveraging and contributing to a |
| 6 shared test suite is one of the most important tools in achieving | 6 shared test suite is one of the most important tools in achieving |
| 7 interoperability between browsers. The [web-platform-tests | 7 interoperability between browsers. The [web-platform-tests |
| 8 repository](https://github.com/w3c/web-platform-tests) is the primary shared | 8 repository](https://github.com/w3c/web-platform-tests) is the primary shared |
| 9 test suite where all browser engines are collaborating. There's also a | 9 test suite where all browser engines are collaborating. There's also a |
| 10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but | 10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but |
| 11 that will [soon be merged into | 11 that will [soon be merged into |
| 12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). | 12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). |
| 13 | 13 |
| 14 Chromium has 2-way import/export process with the upstream web-platform-tests | 14 Chromium has 2-way import/export process with the upstream web-platform-tests |
| 15 repository, where tests are imported into | 15 repository, where tests are imported into |
| 16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) | 16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
| 17 and any changes to the imported tests are also exported to web-platform-tests. | 17 and any changes to the imported tests are also exported to web-platform-tests. |
| 18 | 18 |
| 19 See http://web-platform-tests.org/ for general documentation on |
| 20 web-platform-tests, including tips for writing and reviewing tests. |
| 21 |
| 19 [TOC] | 22 [TOC] |
| 20 | 23 |
| 21 ## Importing tests | 24 ## Importing tests |
| 22 | 25 |
| 23 Chromium has mirrors | 26 Chromium has mirrors |
| 24 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor
m-tests/), | 27 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor
m-tests/), |
| 25 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the | 28 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the |
| 26 GitHub repos, and periodically imports a subset of the tests so that they are | 29 GitHub repos, and periodically imports a subset of the tests so that they are |
| 27 run as part of the regular Blink layout test testing process. | 30 run as part of the regular Blink layout test testing process. |
| 28 | 31 |
| 29 The goal of this process are to be able to run Web Platform Tests unmodified | 32 The goal of this process are to be able to run web-platform-tests unmodified |
| 30 locally just as easily as we can run the Blink tests, and ensure that we are | 33 locally just as easily as we can run the Blink tests, and ensure that we are |
| 31 tracking tip-of-tree in the Web Platform Tests repository as closely as | 34 tracking tip-of-tree in the web-platform-tests repository as closely as |
| 32 possible, and running as many of the tests as possible. | 35 possible, and running as many of the tests as possible. |
| 33 | 36 |
| 34 ### Automatic import process | 37 ### Automatic import process |
| 35 | 38 |
| 36 There is an automatic process for updating the Chromium copy of the Web Platform | 39 There is an automatic process for updating the Chromium copy of |
| 37 Tests. The import is done by the builder [w3c-test-autoroller | 40 web-platform-tests. The import is done by the builder [w3c-test-autoroller |
| 38 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto
roller). | 41 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto
roller). |
| 39 | 42 |
| 40 The easiest way to check the status of recent imports is to look at: | 43 The easiest way to check the status of recent imports is to look at: |
| 41 | 44 |
| 42 - Recent logs on Buildbot for [w3c-test-autoroller | 45 - Recent logs on Buildbot for [w3c-test-autoroller |
| 43 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-
autoroller) | 46 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-
autoroller) |
| 44 - Recent CLs created by | 47 - Recent CLs created by |
| 45 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea
rch?owner=blink-w3c-test-autoroller%40chromium.org). | 48 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea
rch?owner=blink-w3c-test-autoroller%40chromium.org). |
| 46 | 49 |
| 47 Automatic imports are intended to run at least once every 24 hours. | 50 Automatic imports are intended to run at least once every 24 hours. |
| (...skipping 24 matching lines...) Expand all Loading... |
| 72 | 75 |
| 73 If you wish to add more tests (by un-skipping some of the directories currently | 76 If you wish to add more tests (by un-skipping some of the directories currently |
| 74 skipped in `W3CImportExpectations`), you can modify that file locally and commit | 77 skipped in `W3CImportExpectations`), you can modify that file locally and commit |
| 75 it, and on the next auto-import, the new tests should be imported. | 78 it, and on the next auto-import, the new tests should be imported. |
| 76 | 79 |
| 77 If you want to import immediately (in order to try the tests out locally, etc) | 80 If you want to import immediately (in order to try the tests out locally, etc) |
| 78 you can also run `wpt-import --allow-local-commits`, but this is not required. | 81 you can also run `wpt-import --allow-local-commits`, but this is not required. |
| 79 | 82 |
| 80 ## Writing tests | 83 ## Writing tests |
| 81 | 84 |
| 82 To contribute changes to Web Platform Tests, just commit your changes directly | 85 To contribute changes to web-platform-tests, just commit your changes directly |
| 83 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) | 86 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
| 84 and the changes will be automatically upstreamed within 24 hours. | 87 and the changes will be automatically upstreamed within 24 hours. |
| 85 | 88 |
| 86 Changes involving adding, removing or modifying tests can all be upstreamed. | 89 Changes involving adding, removing or modifying tests can all be upstreamed. |
| 87 Any changes outside of | 90 Any changes outside of |
| 88 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be | 91 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be |
| 89 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, | 92 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, |
| 90 will also not be upstreamed. | 93 will also not be upstreamed. |
| 91 | 94 |
| 92 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate | 95 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate |
| 93 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is | 96 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is |
| 94 landed. The command to do so is: | 97 landed. The command to do so is: |
| 95 | 98 |
| 96 ```bash | 99 ```bash |
| 97 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ | 100 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ |
| 98 --tests-root=LayoutTests/external/wpt | 101 --tests-root=LayoutTests/external/wpt |
| 99 ``` | 102 ``` |
| 100 | 103 |
| 101 Most tests are written using testharness.js, see | 104 Most tests are written using testharness.js, see |
| 102 [Writing Layout Tests](./writing_layout_tests.md) and | 105 [Writing Layout Tests](./writing_layout_tests.md) and |
| 103 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines. | 106 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines. |
| 104 | 107 |
| 105 ### Write tests against specifications | 108 ### Write tests against specifications |
| 106 | 109 |
| 107 Tests in Web Platform Tests are expected to match behavior defined by the | 110 Tests in web-platform-tests are expected to match behavior defined by the |
| 108 relevant specification. In other words, all assertions that a test makes | 111 relevant specification. In other words, all assertions that a test makes |
| 109 should be derived from a specification's normative requirements, and not go | 112 should be derived from a specification's normative requirements, and not go |
| 110 beyond them. It is often necessary to change the specification to clarify what | 113 beyond them. It is often necessary to change the specification to clarify what |
| 111 is and isn't required. | 114 is and isn't required. |
| 112 | 115 |
| 113 When the standards discussion is still ongoing or blocked on some implementation | 116 When the standards discussion is still ongoing or blocked on some implementation |
| 114 successfully shipping the hoped-for behavior, write the tests outside of | 117 successfully shipping the hoped-for behavior, write the tests outside of |
| 115 web-platform-tests and upstream them when the specification is finally updated. | 118 web-platform-tests and upstream them when the specification is finally updated. |
| 116 Optionally, it may be possible to write deliberately failing tests against the | 119 Optionally, it may be possible to write deliberately failing tests against the |
| 117 current specification and later update them. | 120 current specification and later update them. |
| (...skipping 30 matching lines...) Expand all Loading... |
| 148 | 151 |
| 149 ### What if there are conflicts? | 152 ### What if there are conflicts? |
| 150 | 153 |
| 151 This cannot be avoided entirely as the two repositories are independent, but | 154 This cannot be avoided entirely as the two repositories are independent, but |
| 152 should be rare with frequent imports and exports. When it does happen, manual | 155 should be rare with frequent imports and exports. When it does happen, manual |
| 153 intervention will be needed and in non-trivial cases you may be asked to help | 156 intervention will be needed and in non-trivial cases you may be asked to help |
| 154 resolve the conflict. | 157 resolve the conflict. |
| 155 | 158 |
| 156 ### Direct pull requests | 159 ### Direct pull requests |
| 157 | 160 |
| 158 It's still possible to make direct pull requests to web-platform-tests. The | 161 It's still possible to make direct pull requests to web-platform-tests, see |
| 159 processes for getting new tests committed the W3C repos are at | 162 http://web-platform-tests.org/appendix/github-intro.html. |
| 160 http://testthewebforward.org/docs/. Some specifics are at | |
| 161 http://testthewebforward.org/docs/github-101.html. | |
| 162 | 163 |
| 163 ## Reviewing tests | 164 ## Reviewing tests |
| 164 | 165 |
| 165 Anyone who can review code and tests in Chromium can also review changes in | 166 Anyone who can review code and tests in Chromium can also review changes in |
| 166 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) | 167 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
| 167 that will be automatically upstreamed. There will be no additional review in | 168 that will be automatically upstreamed. There will be no additional review in |
| 168 web-platform-tests as part of the export process. | 169 web-platform-tests as part of the export process. |
| 169 | 170 |
| 170 If upstream reviewers have feedback on the changes, discuss on the pull request | 171 If upstream reviewers have feedback on the changes, discuss on the pull request |
| 171 created during export, and if necessary work on a new pull request to iterate | 172 created during export, and if necessary work on a new pull request to iterate |
| 172 until everyone is satisfied. | 173 until everyone is satisfied. |
| 173 | 174 |
| 174 When reviewing tests, check that they match the relevant specification, which | 175 When reviewing tests, check that they match the relevant specification, which |
| 175 may not fully match the implementation. See also | 176 may not fully match the implementation. See also |
| 176 [Write tests against specifications](#Write-tests-against-specifications). | 177 [Write tests against specifications](#Write-tests-against-specifications). |
| OLD | NEW |