OLD | NEW |
1 # Web Platform Tests | 1 # Web Platform Tests |
2 | 2 |
3 Interoperability between browsers is | 3 Interoperability between browsers is |
4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's | 4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's |
5 mission of improving the web. We believe that leveraging and contributing to a | 5 mission of improving the web. We believe that leveraging and contributing to a |
6 shared test suite is one of the most important tools in achieving | 6 shared test suite is one of the most important tools in achieving |
7 interoperability between browsers. The [web-platform-tests | 7 interoperability between browsers. The [web-platform-tests |
8 repository](https://github.com/w3c/web-platform-tests) is the primary shared | 8 repository](https://github.com/w3c/web-platform-tests) is the primary shared |
9 test suite where all browser engines are collaborating. There's also a | 9 test suite where all browser engines are collaborating. There's also a |
10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but | 10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but |
11 that will [soon be merged into | 11 that will [soon be merged into |
12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). | 12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). |
13 | 13 |
| 14 Chromium has 2-way import/export process with the upstream web-platform-tests |
| 15 repository, where tests are imported into |
| 16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
| 17 and any changes to the imported tests are also exported to web-platform-tests. |
| 18 |
14 [TOC] | 19 [TOC] |
15 | 20 |
16 ## Import | 21 ## Importing tests |
17 | 22 |
18 Chromium has mirrors | 23 Chromium has mirrors |
19 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor
m-tests/), | 24 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor
m-tests/), |
20 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the | 25 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the |
21 GitHub repos, and periodically imports a subset of the tests so that they are | 26 GitHub repos, and periodically imports a subset of the tests so that they are |
22 run as part of the regular Blink layout test testing process. | 27 run as part of the regular Blink layout test testing process. |
23 | 28 |
24 The goal of this process are to be able to run the Web Platform Tests unmodified | 29 The goal of this process are to be able to run Web Platform Tests unmodified |
25 locally just as easily as we can run the Blink tests, and ensure that we are | 30 locally just as easily as we can run the Blink tests, and ensure that we are |
26 tracking tip-of-tree in the Web Platform Tests repository as closely as | 31 tracking tip-of-tree in the Web Platform Tests repository as closely as |
27 possible, and running as many of the tests as possible. | 32 possible, and running as many of the tests as possible. |
28 | 33 |
29 ### Automatic import process | 34 ### Automatic import process |
30 | 35 |
31 There is an automatic process for updating the Chromium copy of the Web Platform | 36 There is an automatic process for updating the Chromium copy of the Web Platform |
32 Tests. The import is done by the builder [w3c-test-autoroller | 37 Tests. The import is done by the builder [w3c-test-autoroller |
33 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto
roller). | 38 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto
roller). |
34 | 39 |
35 The easiest way to check the status of recent imports is to look at: | 40 The easiest way to check the status of recent imports is to look at: |
36 | 41 |
37 - Recent logs on Buildbot for [w3c-test-autoroller | 42 - Recent logs on Buildbot for [w3c-test-autoroller |
38 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-
autoroller) | 43 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-
autoroller) |
39 - Recent CLs created by | 44 - Recent CLs created by |
40 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea
rch?owner=blink-w3c-test-autoroller%40chromium.org). | 45 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea
rch?owner=blink-w3c-test-autoroller%40chromium.org). |
41 | 46 |
| 47 Automatic imports are intended to run at least once every 24 hours. |
| 48 |
42 ### Skipped tests | 49 ### Skipped tests |
43 | 50 |
44 We control which tests are imported via a file called | 51 We control which tests are imported via a file called |
45 [W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectatio
ns), | 52 [W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectatio
ns), |
46 which has a list of directories to skip while importing. | 53 which has a list of directories to skip while importing. |
47 | 54 |
48 In addition to the directories and tests explicitly skipped there, tests may | 55 In addition to the directories and tests explicitly skipped there, tests may |
49 also be skipped for a couple other reasons, e.g. if the file path is too long | 56 also be skipped for a couple other reasons, e.g. if the file path is too long |
50 for Windows. To check what files are skipped in import, check the recent logs | 57 for Windows. To check what files are skipped in import, check the recent logs |
51 for [w3c-test-autoroller | 58 for [w3c-test-autoroller |
(...skipping 11 matching lines...) Expand all Loading... |
63 | 70 |
64 ### Enabling import for a new directory | 71 ### Enabling import for a new directory |
65 | 72 |
66 If you wish to add more tests (by un-skipping some of the directories currently | 73 If you wish to add more tests (by un-skipping some of the directories currently |
67 skipped in `W3CImportExpectations`), you can modify that file locally and commit | 74 skipped in `W3CImportExpectations`), you can modify that file locally and commit |
68 it, and on the next auto-import, the new tests should be imported. | 75 it, and on the next auto-import, the new tests should be imported. |
69 | 76 |
70 If you want to import immediately (in order to try the tests out locally, etc) | 77 If you want to import immediately (in order to try the tests out locally, etc) |
71 you can also run `wpt-import --allow-local-commits`, but this is not required. | 78 you can also run `wpt-import --allow-local-commits`, but this is not required. |
72 | 79 |
73 ## Contributing tests back to the Web Platform Tests project. | 80 ## Writing tests |
74 | 81 |
75 If you need to make changes to Web Platform Tests, just commit your changes | 82 To contribute changes to Web Platform Tests, just commit your changes directly |
76 directly to | 83 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
77 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) | |
78 and the changes will be automatically upstreamed within 24 hours. | 84 and the changes will be automatically upstreamed within 24 hours. |
79 | 85 |
80 Note that tests in Web Platform Tests are expected to match behavior defined by | 86 Changes involving adding, removing or modifying tests can all be upstreamed. |
81 the relevant WHATWG or W3C specification, not simply Blink's behavior. If in | 87 Any changes outside of |
82 doubt, please request code review from someone with expertise in the relevant | 88 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be |
83 specification text. | 89 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, |
| 90 will also not be upstreamed. |
84 | 91 |
85 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate | 92 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate |
86 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is | 93 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is |
87 landed. The command to do so is: | 94 landed. The command to do so is: |
88 | 95 |
89 ```bash | 96 ```bash |
90 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ | 97 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ |
91 --tests-root=LayoutTests/external/wpt | 98 --tests-root=LayoutTests/external/wpt |
92 ``` | 99 ``` |
93 | 100 |
94 ### What kinds of changes can be upstreamed? | 101 Most tests are written using testharness.js, see |
| 102 [Writing Layout Tests](./writing_layout_tests.md) and |
| 103 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines. |
95 | 104 |
96 In general, changes involving adding, removing or modifying tests can all be | 105 ### Write tests against specifications |
97 upstreamed. From a Chromium commit, any changes outside of | 106 |
98 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be | 107 Tests in Web Platform Tests are expected to match behavior defined by the |
99 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, | 108 relevant specification. In other words, all assertions that a test makes |
100 will also not be upstreamed. | 109 should be derived from a specification's normative requirements, and not go |
| 110 beyond them. It is often necessary to change the specification to clarify what |
| 111 is and isn't required. |
| 112 |
| 113 When the standards discussion is still ongoing or blocked on some implementation |
| 114 successfully shipping the hoped-for behavior, write the tests outside of |
| 115 web-platform-tests and upstream them when the specification is finally updated. |
| 116 Optionally, it may be possible to write deliberately failing tests against the |
| 117 current specification and later update them. |
| 118 |
| 119 ### Tests that require testing APIs |
| 120 |
| 121 Tests that depend on `internals.*`, `eventSender.*` or other internal testing |
| 122 APIs cannot yet be written as part of web-platform-tests. |
| 123 |
| 124 An alternative is to write manual tests that are automated with scripts from |
| 125 [wpt_automation](../../third_party/WebKit/LayoutTests/external/wpt_automation). |
| 126 Such tests still require case-by-case automation to run for other browser |
| 127 engines, but are more valuable than purely manual tests. |
| 128 |
| 129 *** note |
| 130 TODO(foolip): Figure out and document a more scalable test automation solution. |
| 131 *** |
| 132 |
| 133 ### Adding new top-level directories |
101 | 134 |
102 Entirely new top-level directories should generally be added upstream, since | 135 Entirely new top-level directories should generally be added upstream, since |
103 that's the only way to add an OWNERS file upstream. After adding a new top-level | 136 that's the only way to add an OWNERS file upstream. After adding a new top-level |
104 directory upstream, you should add a line for it in `W3CImportExpectations`. | 137 directory upstream, you should add a line for it in `W3CImportExpectations`. |
105 | 138 |
| 139 Adding the new directory (and `W3CImportExpectations` entry) in Chromium and |
| 140 later adding an OWNERS file upstream also works. |
| 141 |
106 ### Will the exported commits be linked to my GitHub profile? | 142 ### Will the exported commits be linked to my GitHub profile? |
107 | 143 |
108 The email you commit with in Chromium will be the author of the commit on | 144 The email you commit with in Chromium will be the author of the commit on |
109 GitHub. You can [add it as a secondary address on your GitHub | 145 GitHub. You can [add it as a secondary address on your GitHub |
110 account](https://help.github.com/articles/adding-an-email-address-to-your-github
-account/) | 146 account](https://help.github.com/articles/adding-an-email-address-to-your-github
-account/) |
111 to link your exported commits to your GitHub profile. | 147 to link your exported commits to your GitHub profile. |
112 | 148 |
113 ### What if there are conflicts? | 149 ### What if there are conflicts? |
114 | 150 |
115 This cannot be avoided entirely as the two repositories are independent, but | 151 This cannot be avoided entirely as the two repositories are independent, but |
116 should be rare with frequent imports and exports. When it does happen, manual | 152 should be rare with frequent imports and exports. When it does happen, manual |
117 intervention will be needed and in non-trivial cases you may be asked to help | 153 intervention will be needed and in non-trivial cases you may be asked to help |
118 resolve the conflict. | 154 resolve the conflict. |
119 | 155 |
120 ### Direct pull requests | 156 ### Direct pull requests |
121 | 157 |
122 It's still possible to make direct pull requests to web-platform-tests. The | 158 It's still possible to make direct pull requests to web-platform-tests. The |
123 processes for getting new tests committed the W3C repos are at | 159 processes for getting new tests committed the W3C repos are at |
124 http://testthewebforward.org/docs/. Some specifics are at | 160 http://testthewebforward.org/docs/. Some specifics are at |
125 http://testthewebforward.org/docs/github-101.html. | 161 http://testthewebforward.org/docs/github-101.html. |
| 162 |
| 163 ## Reviewing tests |
| 164 |
| 165 Anyone who can review code and tests in Chromium can also review changes in |
| 166 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) |
| 167 that will be automatically upstreamed. There will be no additional review in |
| 168 web-platform-tests as part of the export process. |
| 169 |
| 170 If upstream reviewers have feedback on the changes, discuss on the pull request |
| 171 created during export, and if necessary work on a new pull request to iterate |
| 172 until everyone is satisfied. |
| 173 |
| 174 When reviewing tests, check that they match the relevant specification, which |
| 175 may not fully match the implementation. See also |
| 176 [Write tests against specifications](#Write-tests-against-specifications). |
OLD | NEW |