Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(219)

Side by Side Diff: docs/testing/web_platform_tests.md

Issue 2695963002: Update web-platform-tests docs (Closed)
Patch Set: Created 3 years, 10 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « docs/testing/test_browser_dialog.md ('k') | docs/testing/writing_layout_tests.md » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Web Platform Tests 1 # Web Platform Tests
2 2
3 Interoperability between browsers is 3 Interoperability between browsers is
4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's 4 [critical](https://www.chromium.org/blink/platform-predictability) to Chromium's
5 mission of improving the web. We believe that leveraging and contributing to a 5 mission of improving the web. We believe that leveraging and contributing to a
6 shared test suite is one of the most important tools in achieving 6 shared test suite is one of the most important tools in achieving
7 interoperability between browsers. The [web-platform-tests 7 interoperability between browsers. The [web-platform-tests
8 repository](https://github.com/w3c/web-platform-tests) is the primary shared 8 repository](https://github.com/w3c/web-platform-tests) is the primary shared
9 test suite where all browser engines are collaborating. There's also a 9 test suite where all browser engines are collaborating. There's also a
10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but 10 [csswg-test repository](https://github.com/w3c/csswg-test) for CSS tests, but
11 that will [soon be merged into 11 that will [soon be merged into
12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102). 12 web-platform-tests](https://github.com/w3c/csswg-test/issues/1102).
13 13
14 Chromium has 2-way import/export process with the upstream web-platform-tests
15 repository, where tests are imported into
16 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
17 and any changes to the imported tests are also exported to web-platform-tests.
18
14 [TOC] 19 [TOC]
15 20
16 ## Import 21 ## Importing tests
17 22
18 Chromium has mirrors 23 Chromium has mirrors
19 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor m-tests/), 24 ([web-platform-tests](https://chromium.googlesource.com/external/w3c/web-platfor m-tests/),
20 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the 25 [csswg-test](https://chromium.googlesource.com/external/w3c/csswg-test/)) of the
21 GitHub repos, and periodically imports a subset of the tests so that they are 26 GitHub repos, and periodically imports a subset of the tests so that they are
22 run as part of the regular Blink layout test testing process. 27 run as part of the regular Blink layout test testing process.
23 28
24 The goal of this process are to be able to run the Web Platform Tests unmodified 29 The goal of this process are to be able to run Web Platform Tests unmodified
25 locally just as easily as we can run the Blink tests, and ensure that we are 30 locally just as easily as we can run the Blink tests, and ensure that we are
26 tracking tip-of-tree in the Web Platform Tests repository as closely as 31 tracking tip-of-tree in the Web Platform Tests repository as closely as
27 possible, and running as many of the tests as possible. 32 possible, and running as many of the tests as possible.
28 33
29 ### Automatic import process 34 ### Automatic import process
30 35
31 There is an automatic process for updating the Chromium copy of the Web Platform 36 There is an automatic process for updating the Chromium copy of the Web Platform
32 Tests. The import is done by the builder [w3c-test-autoroller 37 Tests. The import is done by the builder [w3c-test-autoroller
33 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto roller). 38 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-auto roller).
34 39
35 The easiest way to check the status of recent imports is to look at: 40 The easiest way to check the status of recent imports is to look at:
36 41
37 - Recent logs on Buildbot for [w3c-test-autoroller 42 - Recent logs on Buildbot for [w3c-test-autoroller
38 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test- autoroller) 43 builder](https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test- autoroller)
39 - Recent CLs created by 44 - Recent CLs created by
40 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea rch?owner=blink-w3c-test-autoroller%40chromium.org). 45 [blink-w3c-test-autoroller@chromium.org](https://codereview.chromium.org/sea rch?owner=blink-w3c-test-autoroller%40chromium.org).
41 46
47 Automatic imports are intended to run at least once every 24 hours.
48
42 ### Skipped tests 49 ### Skipped tests
43 50
44 We control which tests are imported via a file called 51 We control which tests are imported via a file called
45 [W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectatio ns), 52 [W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectatio ns),
46 which has a list of directories to skip while importing. 53 which has a list of directories to skip while importing.
47 54
48 In addition to the directories and tests explicitly skipped there, tests may 55 In addition to the directories and tests explicitly skipped there, tests may
49 also be skipped for a couple other reasons, e.g. if the file path is too long 56 also be skipped for a couple other reasons, e.g. if the file path is too long
50 for Windows. To check what files are skipped in import, check the recent logs 57 for Windows. To check what files are skipped in import, check the recent logs
51 for [w3c-test-autoroller 58 for [w3c-test-autoroller
(...skipping 11 matching lines...) Expand all
63 70
64 ### Enabling import for a new directory 71 ### Enabling import for a new directory
65 72
66 If you wish to add more tests (by un-skipping some of the directories currently 73 If you wish to add more tests (by un-skipping some of the directories currently
67 skipped in `W3CImportExpectations`), you can modify that file locally and commit 74 skipped in `W3CImportExpectations`), you can modify that file locally and commit
68 it, and on the next auto-import, the new tests should be imported. 75 it, and on the next auto-import, the new tests should be imported.
69 76
70 If you want to import immediately (in order to try the tests out locally, etc) 77 If you want to import immediately (in order to try the tests out locally, etc)
71 you can also run `wpt-import --allow-local-commits`, but this is not required. 78 you can also run `wpt-import --allow-local-commits`, but this is not required.
72 79
73 ## Contributing tests back to the Web Platform Tests project. 80 ## Writing tests
74 81
75 If you need to make changes to Web Platform Tests, just commit your changes 82 To contribute changes to Web Platform Tests, just commit your changes directly
76 directly to 83 to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
77 [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
78 and the changes will be automatically upstreamed within 24 hours. 84 and the changes will be automatically upstreamed within 24 hours.
79 85
80 Note that tests in Web Platform Tests are expected to match behavior defined by 86 Changes involving adding, removing or modifying tests can all be upstreamed.
81 the relevant WHATWG or W3C specification, not simply Blink's behavior. If in 87 Any changes outside of
82 doubt, please request code review from someone with expertise in the relevant 88 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be
83 specification text. 89 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
90 will also not be upstreamed.
84 91
85 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate 92 Note: if you're adding a new test in `external/wpt`, you'll need to re-generate
86 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is 93 MANIFEST.json manually until [CL 2644783003](https://crrev.com/2644783003) is
87 landed. The command to do so is: 94 landed. The command to do so is:
88 95
89 ```bash 96 ```bash
90 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \ 97 Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \
91 --tests-root=LayoutTests/external/wpt 98 --tests-root=LayoutTests/external/wpt
92 ``` 99 ```
93 100
94 ### What kinds of changes can be upstreamed? 101 Most tests are written using testharness.js, see
102 [Writing Layout Tests](./writing_layout_tests.md) and
103 [Layout Tests Tips](./layout_tests_tips.md) for general guidelines.
95 104
96 In general, changes involving adding, removing or modifying tests can all be 105 ### Write tests against specifications
97 upstreamed. From a Chromium commit, any changes outside of 106
98 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be 107 Tests in Web Platform Tests are expected to match behavior defined by the
99 upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, 108 relevant specification. In other words, all assertions that a test makes
100 will also not be upstreamed. 109 should be derived from a specification's normative requirements, and not go
110 beyond them. It is often necessary to change the specification to clarify what
111 is and isn't required.
112
113 When the standards discussion is still ongoing or blocked on some implementation
114 successfully shipping the hoped-for behavior, write the tests outside of
115 web-platform-tests and upstream them when the specification is finally updated.
116 Optionally, it may be possible to write deliberately failing tests against the
117 current specification and later update them.
118
119 ### Tests that require testing APIs
120
121 Tests that depend on `internals.*`, `eventSender.*` or other internal testing
122 APIs cannot yet be written as part of web-platform-tests.
123
124 An alternative is to write manual tests that are automated with scripts from
125 [wpt_automation](../../third_party/WebKit/LayoutTests/external/wpt_automation).
126 Such tests still require case-by-case automation to run for other browser
127 engines, but are more valuable than purely manual tests.
128
129 *** note
130 TODO(foolip): Figure out and document a more scalable test automation solution.
131 ***
132
133 ### Adding new top-level directories
101 134
102 Entirely new top-level directories should generally be added upstream, since 135 Entirely new top-level directories should generally be added upstream, since
103 that's the only way to add an OWNERS file upstream. After adding a new top-level 136 that's the only way to add an OWNERS file upstream. After adding a new top-level
104 directory upstream, you should add a line for it in `W3CImportExpectations`. 137 directory upstream, you should add a line for it in `W3CImportExpectations`.
105 138
139 Adding the new directory (and `W3CImportExpectations` entry) in Chromium and
140 later adding an OWNERS file upstream also works.
141
106 ### Will the exported commits be linked to my GitHub profile? 142 ### Will the exported commits be linked to my GitHub profile?
107 143
108 The email you commit with in Chromium will be the author of the commit on 144 The email you commit with in Chromium will be the author of the commit on
109 GitHub. You can [add it as a secondary address on your GitHub 145 GitHub. You can [add it as a secondary address on your GitHub
110 account](https://help.github.com/articles/adding-an-email-address-to-your-github -account/) 146 account](https://help.github.com/articles/adding-an-email-address-to-your-github -account/)
111 to link your exported commits to your GitHub profile. 147 to link your exported commits to your GitHub profile.
112 148
113 ### What if there are conflicts? 149 ### What if there are conflicts?
114 150
115 This cannot be avoided entirely as the two repositories are independent, but 151 This cannot be avoided entirely as the two repositories are independent, but
116 should be rare with frequent imports and exports. When it does happen, manual 152 should be rare with frequent imports and exports. When it does happen, manual
117 intervention will be needed and in non-trivial cases you may be asked to help 153 intervention will be needed and in non-trivial cases you may be asked to help
118 resolve the conflict. 154 resolve the conflict.
119 155
120 ### Direct pull requests 156 ### Direct pull requests
121 157
122 It's still possible to make direct pull requests to web-platform-tests. The 158 It's still possible to make direct pull requests to web-platform-tests. The
123 processes for getting new tests committed the W3C repos are at 159 processes for getting new tests committed the W3C repos are at
124 http://testthewebforward.org/docs/. Some specifics are at 160 http://testthewebforward.org/docs/. Some specifics are at
125 http://testthewebforward.org/docs/github-101.html. 161 http://testthewebforward.org/docs/github-101.html.
162
163 ## Reviewing tests
164
165 Anyone who can review code and tests in Chromium can also review changes in
166 [external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
167 that will be automatically upstreamed. There will be no additional review in
168 web-platform-tests as part of the export process.
169
170 If upstream reviewers have feedback on the changes, discuss on the pull request
171 created during export, and if necessary work on a new pull request to iterate
172 until everyone is satisfied.
173
174 When reviewing tests, check that they match the relevant specification, which
175 may not fully match the implementation. See also
176 [Write tests against specifications](#Write-tests-against-specifications).
OLDNEW
« no previous file with comments | « docs/testing/test_browser_dialog.md ('k') | docs/testing/writing_layout_tests.md » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698