OLD | NEW |
1 # Web Platform Tests | 1 # web-platform-tests |
2 | 2 |
3 Interoperability between browsers is | 3 Interoperability between browsers is |
4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's | 4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's |
5 mission of improving the web. We believe that leveraging and contributing to a | 5 mission of improving the web. We believe that leveraging and contributing to a |
6 shared test suite is one of the most important tools in achieving | 6 shared test suite is one of the most important tools in achieving |
7 interoperability between browsers. The [web-platform-tests | 7 interoperability between browsers. The [web-platform-tests |
8 repository](https://github.com/w3c/web-platform-tests) is the primary shared | 8 repository](https://github.com/w3c/web-platform-tests) is the primary shared |
9 test suite where all browser engines are collaborating. There's also a | 9 test suite where all browser engines are collaborating. There's also a |
10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but | 10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but |
11 that will [soon be merged into | 11 that will [soon be merged into |
12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). | 12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). |
13 | 13 |
14 Chromium has 2-way import/export process with the upstream web-platform-tests | 14 Chromium has 2-way import/export process with the upstream web-platform-tests |
15 repository, where tests are imported into | 15 repository, where tests are imported into |
16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) | 16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
17 and any changes to the imported tests are also exported to web-platform-tests. | 17 and any changes to the imported tests are also exported to web-platform-tests. |
18 | 18 |
| 19 See http://web-platform-tests.org/ for general documentation on |
| 20 web-platform-tests, including tips for writing and reviewing tests. |
| 21 |
19 [TOC] | 22 [TOC] |
20 | 23 |
21 ## Importing tests | 24 ## Importing tests |
22 | 25 |
23 Chromium has mirrors | 26 Chromium has mirrors |
24 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor
m-tests/), | 27 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor
m-tests/), |
25 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the | 28 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the |
26 GitHub repos, and periodically imports a subset of the tests so that they are | 29 GitHub repos, and periodically imports a subset of the tests so that they are |
27 run as part of the regular Blink layout test testing process. | 30 run as part of the regular Blink layout test testing process. |
28 | 31 |
29 The goal of this process are to be able to run Web Platform Tests unmodified | 32 The goal of this process are to be able to run web-platform-tests unmodified |
30 locally just as easily as we can run the Blink tests, and ensure that we are | 33 locally just as easily as we can run the Blink tests, and ensure that we are |
31 tracking tip-of-tree in the Web Platform Tests repository as closely as | 34 tracking tip-of-tree in the web-platform-tests repository as closely as |
32 possible, and running as many of the tests as possible. | 35 possible, and running as many of the tests as possible. |
33 | 36 |
34 ### Automatic import process | 37 ### Automatic import process |
35 | 38 |
36 There is an automatic process for updating the Chromium copy of the Web Platform | 39 There is an automatic process for updating the Chromium copy of |
37 Tests. The import is done by the builder [w3c-test-autoroller | 40 web-platform-tests. The import is done by the builder [w3c-test-autoroller |
38 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto
roller). | 41 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto
roller). |
39 | 42 |
40 The easiest way to check the status of recent imports is to look at: | 43 The easiest way to check the status of recent imports is to look at: |
41 | 44 |
42 - Recent logs on Buildbot for [w3c-test-autoroller | 45 - Recent logs on Buildbot for [w3c-test-autoroller |
43 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-
autoroller) | 46 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-
autoroller) |
44 - Recent CLs created by | 47 - Recent CLs created by |
45 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea
rch?owner=blink-w3c-test-autoroller%40chromium.org). | 48 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea
rch?owner=blink-w3c-test-autoroller%40chromium.org). |
46 | 49 |
47 Automatic imports are intended to run at least once every 24 hours. | 50 Automatic imports are intended to run at least once every 24 hours. |
(...skipping 24 matching lines...) Expand all Loading... |
72 | 75 |
73 If you wish to add more tests (by un-skipping some of the directories currently | 76 If you wish to add more tests (by un-skipping some of the directories currently |
74 skipped in `W3CImportExpectations`), you can modify that file locally and commit | 77 skipped in `W3CImportExpectations`), you can modify that file locally and commit |
75 it, and on the next auto-import, the new tests should be imported. | 78 it, and on the next auto-import, the new tests should be imported. |
76 | 79 |
77 If you want to import immediately (in order to try the tests out locally, etc) | 80 If you want to import immediately (in order to try the tests out locally, etc) |
78 you can also run `wpt-import --allow-local-commits`, but this is not required. | 81 you can also run `wpt-import --allow-local-commits`, but this is not required. |
79 | 82 |
80 ## Writing tests | 83 ## Writing tests |
81 | 84 |
82 To contribute changes to Web Platform Tests, just commit your changes directly | 85 To contribute changes to web-platform-tests, just commit your changes directly |
83 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) | 86 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
84 and the changes will be automatically upstreamed within 24 hours. | 87 and the changes will be automatically upstreamed within 24 hours. |
85 | 88 |
86 Changes involving adding, removing or modifying tests can all be upstreamed. | 89 Changes involving adding, removing or modifying tests can all be upstreamed. |
87 Any changes outside of | 90 Any changes outside of |
88 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be | 91 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be |
89 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, | 92 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, |
90 will also not be upstreamed. | 93 will also not be upstreamed. |
91 | 94 |
92 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate | 95 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate |
93 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is | 96 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is |
94 landed. The command to do so is: | 97 landed. The command to do so is: |
95 | 98 |
96 ```bash | 99 ```bash |
97 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ | 100 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ |
98 --tests-root=LayoutTests/external/wpt | 101 --tests-root=LayoutTests/external/wpt |
99 ``` | 102 ``` |
100 | 103 |
101 Most tests are written using testharness.js, see | 104 Most tests are written using testharness.js, see |
102 [Writing Layout Tests](./writing_layout_tests.md) and | 105 [Writing Layout Tests](./writing_layout_tests.md) and |
103 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines. | 106 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines. |
104 | 107 |
105 ### Write tests against specifications | 108 ### Write tests against specifications |
106 | 109 |
107 Tests in Web Platform Tests are expected to match behavior defined by the | 110 Tests in web-platform-tests are expected to match behavior defined by the |
108 relevant specification. In other words, all assertions that a test makes | 111 relevant specification. In other words, all assertions that a test makes |
109 should be derived from a specification's normative requirements, and not go | 112 should be derived from a specification's normative requirements, and not go |
110 beyond them. It is often necessary to change the specification to clarify what | 113 beyond them. It is often necessary to change the specification to clarify what |
111 is and isn't required. | 114 is and isn't required. |
112 | 115 |
113 When the standards discussion is still ongoing or blocked on some implementation | 116 When the standards discussion is still ongoing or blocked on some implementation |
114 successfully shipping the hoped-for behavior, write the tests outside of | 117 successfully shipping the hoped-for behavior, write the tests outside of |
115 web-platform-tests and upstream them when the specification is finally updated. | 118 web-platform-tests and upstream them when the specification is finally updated. |
116 Optionally, it may be possible to write deliberately failing tests against the | 119 Optionally, it may be possible to write deliberately failing tests against the |
117 current specification and later update them. | 120 current specification and later update them. |
(...skipping 30 matching lines...) Expand all Loading... |
148 | 151 |
149 ### What if there are conflicts? | 152 ### What if there are conflicts? |
150 | 153 |
151 This cannot be avoided entirely as the two repositories are independent, but | 154 This cannot be avoided entirely as the two repositories are independent, but |
152 should be rare with frequent imports and exports. When it does happen, manual | 155 should be rare with frequent imports and exports. When it does happen, manual |
153 intervention will be needed and in non-trivial cases you may be asked to help | 156 intervention will be needed and in non-trivial cases you may be asked to help |
154 resolve the conflict. | 157 resolve the conflict. |
155 | 158 |
156 ### Direct pull requests | 159 ### Direct pull requests |
157 | 160 |
158 It's still possible to make direct pull requests to web-platform-tests. The | 161 It's still possible to make direct pull requests to web-platform-tests, see |
159 processes for getting new tests committed the W3C repos are at | 162 http://web-platform-tests.org/appendix/github-intro.html. |
160 http://testthewebforward.org/docs/. Some specifics are at | |
161 http://testthewebforward.org/docs/github-101.html. | |
162 | 163 |
163 ## Reviewing tests | 164 ## Reviewing tests |
164 | 165 |
165 Anyone who can review code and tests in Chromium can also review changes in | 166 Anyone who can review code and tests in Chromium can also review changes in |
166 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) | 167 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
167 that will be automatically upstreamed. There will be no additional review in | 168 that will be automatically upstreamed. There will be no additional review in |
168 web-platform-tests as part of the export process. | 169 web-platform-tests as part of the export process. |
169 | 170 |
170 If upstream reviewers have feedback on the changes, discuss on the pull request | 171 If upstream reviewers have feedback on the changes, discuss on the pull request |
171 created during export, and if necessary work on a new pull request to iterate | 172 created during export, and if necessary work on a new pull request to iterate |
172 until everyone is satisfied. | 173 until everyone is satisfied. |
173 | 174 |
174 When reviewing tests, check that they match the relevant specification, which | 175 When reviewing tests, check that they match the relevant specification, which |
175 may not fully match the implementation. See also | 176 may not fully match the implementation. See also |
176 [Write tests against specifications](#Write-tests-against-specifications). | 177 [Write tests against specifications](#Write-tests-against-specifications). |
OLD | NEW |