Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(87)

Side by Side Diff: docs/testing/web_platform_tests.md

Issue 2696323003: Link to new web-platform-tests.org documentation from our own (Closed)
Patch Set: Created 3 years, 10 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
OLDNEW
1 # Web Platform Tests 1 # web-platform-tests
2 2
3 Interoperability between browsers is 3 Interoperability between browsers is
4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's 4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's
5 mission of improving the web. We believe that leveraging and contributing to a 5 mission of improving the web. We believe that leveraging and contributing to a
6 shared test suite is one of the most important tools in achieving 6 shared test suite is one of the most important tools in achieving
7 interoperability between browsers. The [web-platform-tests 7 interoperability between browsers. The [web-platform-tests
8 repository](https://github.com/w3c/web-platform-tests) is the primary shared 8 repository](https://github.com/w3c/web-platform-tests) is the primary shared
9 test suite where all browser engines are collaborating. There's also a 9 test suite where all browser engines are collaborating. There's also a
10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but 10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but
11 that will [soon be merged into 11 that will [soon be merged into
12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). 12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102).
13 13
14 Chromium has 2-way import/export process with the upstream web-platform-tests 14 Chromium has 2-way import/export process with the upstream web-platform-tests
15 repository, where tests are imported into 15 repository, where tests are imported into
16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) 16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
17 and any changes to the imported tests are also exported to web-platform-tests. 17 and any changes to the imported tests are also exported to web-platform-tests.
18 18
19 See http://web-platform-tests.org/ for general documentation on
20 web-platform-tests, including tips for writing and reviewing tests.
21
19 [TOC] 22 [TOC]
20 23
21 ## Importing tests 24 ## Importing tests
22 25
23 Chromium has mirrors 26 Chromium has mirrors
24 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor m-tests/), 27 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor m-tests/),
25 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the 28 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the
26 GitHub repos, and periodically imports a subset of the tests so that they are 29 GitHub repos, and periodically imports a subset of the tests so that they are
27 run as part of the regular Blink layout test testing process. 30 run as part of the regular Blink layout test testing process.
28 31
29 The goal of this process are to be able to run Web Platform Tests unmodified 32 The goal of this process are to be able to run web-platform-tests unmodified
30 locally just as easily as we can run the Blink tests, and ensure that we are 33 locally just as easily as we can run the Blink tests, and ensure that we are
31 tracking tip-of-tree in the Web Platform Tests repository as closely as 34 tracking tip-of-tree in the web-platform-tests repository as closely as
32 possible, and running as many of the tests as possible. 35 possible, and running as many of the tests as possible.
33 36
34 ### Automatic import process 37 ### Automatic import process
35 38
36 There is an automatic process for updating the Chromium copy of the Web Platform 39 There is an automatic process for updating the Chromium copy of
37 Tests. The import is done by the builder [w3c-test-autoroller 40 web-platform-tests. The import is done by the builder [w3c-test-autoroller
38 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto roller). 41 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto roller).
39 42
40 The easiest way to check the status of recent imports is to look at: 43 The easiest way to check the status of recent imports is to look at:
41 44
42 - Recent logs on Buildbot for [w3c-test-autoroller 45 - Recent logs on Buildbot for [w3c-test-autoroller
43 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test- autoroller) 46 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test- autoroller)
44 - Recent CLs created by 47 - Recent CLs created by
45 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea rch?owner=blink-w3c-test-autoroller%40chromium.org). 48 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea rch?owner=blink-w3c-test-autoroller%40chromium.org).
46 49
47 Automatic imports are intended to run at least once every 24 hours. 50 Automatic imports are intended to run at least once every 24 hours.
(...skipping 24 matching lines...) Expand all
72 75
73 If you wish to add more tests (by un-skipping some of the directories currently 76 If you wish to add more tests (by un-skipping some of the directories currently
74 skipped in `W3CImportExpectations`), you can modify that file locally and commit 77 skipped in `W3CImportExpectations`), you can modify that file locally and commit
75 it, and on the next auto-import, the new tests should be imported. 78 it, and on the next auto-import, the new tests should be imported.
76 79
77 If you want to import immediately (in order to try the tests out locally, etc) 80 If you want to import immediately (in order to try the tests out locally, etc)
78 you can also run `wpt-import --allow-local-commits`, but this is not required. 81 you can also run `wpt-import --allow-local-commits`, but this is not required.
79 82
80 ## Writing tests 83 ## Writing tests
81 84
82 To contribute changes to Web Platform Tests, just commit your changes directly 85 To contribute changes to web-platform-tests, just commit your changes directly
83 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) 86 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
84 and the changes will be automatically upstreamed within 24 hours. 87 and the changes will be automatically upstreamed within 24 hours.
85 88
86 Changes involving adding, removing or modifying tests can all be upstreamed. 89 Changes involving adding, removing or modifying tests can all be upstreamed.
87 Any changes outside of 90 Any changes outside of
88 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be 91 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be
89 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, 92 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
90 will also not be upstreamed. 93 will also not be upstreamed.
91 94
92 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate 95 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate
93 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is 96 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is
94 landed. The command to do so is: 97 landed. The command to do so is:
95 98
96 ```bash 99 ```bash
97 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ 100 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \
98 --tests-root=LayoutTests/external/wpt 101 --tests-root=LayoutTests/external/wpt
99 ``` 102 ```
100 103
101 Most tests are written using testharness.js, see 104 Most tests are written using testharness.js, see
102 [Writing Layout Tests](./writing_layout_tests.md) and 105 [Writing Layout Tests](./writing_layout_tests.md) and
103 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines. 106 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines.
104 107
105 ### Write tests against specifications 108 ### Write tests against specifications
106 109
107 Tests in Web Platform Tests are expected to match behavior defined by the 110 Tests in web-platform-tests are expected to match behavior defined by the
108 relevant specification. In other words, all assertions that a test makes 111 relevant specification. In other words, all assertions that a test makes
109 should be derived from a specification's normative requirements, and not go 112 should be derived from a specification's normative requirements, and not go
110 beyond them. It is often necessary to change the specification to clarify what 113 beyond them. It is often necessary to change the specification to clarify what
111 is and isn't required. 114 is and isn't required.
112 115
113 When the standards discussion is still ongoing or blocked on some implementation 116 When the standards discussion is still ongoing or blocked on some implementation
114 successfully shipping the hoped-for behavior, write the tests outside of 117 successfully shipping the hoped-for behavior, write the tests outside of
115 web-platform-tests and upstream them when the specification is finally updated. 118 web-platform-tests and upstream them when the specification is finally updated.
116 Optionally, it may be possible to write deliberately failing tests against the 119 Optionally, it may be possible to write deliberately failing tests against the
117 current specification and later update them. 120 current specification and later update them.
(...skipping 30 matching lines...) Expand all
148 151
149 ### What if there are conflicts? 152 ### What if there are conflicts?
150 153
151 This cannot be avoided entirely as the two repositories are independent, but 154 This cannot be avoided entirely as the two repositories are independent, but
152 should be rare with frequent imports and exports. When it does happen, manual 155 should be rare with frequent imports and exports. When it does happen, manual
153 intervention will be needed and in non-trivial cases you may be asked to help 156 intervention will be needed and in non-trivial cases you may be asked to help
154 resolve the conflict. 157 resolve the conflict.
155 158
156 ### Direct pull requests 159 ### Direct pull requests
157 160
158 It's still possible to make direct pull requests to web-platform-tests. The 161 It's still possible to make direct pull requests to web-platform-tests, see
159 processes for getting new tests committed the W3C repos are at 162 http://web-platform-tests.org/appendix/github-intro.html.
160 http://testthewebforward.org/docs/. Some specifics are at
161 http://testthewebforward.org/docs/github-101.html.
162 163
163 ## Reviewing tests 164 ## Reviewing tests
164 165
165 Anyone who can review code and tests in Chromium can also review changes in 166 Anyone who can review code and tests in Chromium can also review changes in
166 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) 167 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
167 that will be automatically upstreamed. There will be no additional review in 168 that will be automatically upstreamed. There will be no additional review in
168 web-platform-tests as part of the export process. 169 web-platform-tests as part of the export process.
169 170
170 If upstream reviewers have feedback on the changes, discuss on the pull request 171 If upstream reviewers have feedback on the changes, discuss on the pull request
171 created during export, and if necessary work on a new pull request to iterate 172 created during export, and if necessary work on a new pull request to iterate
172 until everyone is satisfied. 173 until everyone is satisfied.
173 174
174 When reviewing tests, check that they match the relevant specification, which 175 When reviewing tests, check that they match the relevant specification, which
175 may not fully match the implementation. See also 176 may not fully match the implementation. See also
176 [Write tests against specifications](#Write-tests-against-specifications). 177 [Write tests against specifications](#Write-tests-against-specifications).
OLDNEW

Powered by Google App Engine
This is Rietveld 408576698