OLD | NEW |
1 # Writing Layout Tests | 1 # Writing Layout Tests |
2 | 2 |
3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), |
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
6 or XHTML) and lives in | 6 or XHTML) and lives in |
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
8 | 8 |
9 [TOC] | 9 [TOC] |
10 | 10 |
(...skipping 13 matching lines...) Expand all Loading... |
24 cannot be easily covered by | 24 cannot be easily covered by |
25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour
ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), | 25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour
ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), |
26 the feature must be covered by layout tests, to avoid unexpected regressions. | 26 the feature must be covered by layout tests, to avoid unexpected regressions. |
27 These tests will use Blink-specific testing APIs that are only available in | 27 These tests will use Blink-specific testing APIs that are only available in |
28 [content_shell](./layout_tests_in_content_shell.md). | 28 [content_shell](./layout_tests_in_content_shell.md). |
29 | 29 |
30 *** promo | 30 *** promo |
31 If you know that Blink layout tests are upstreamed to other projects, such as | 31 If you know that Blink layout tests are upstreamed to other projects, such as |
32 [test262](https://github.com/tc39/test262), please update this document. Most | 32 [test262](https://github.com/tc39/test262), please update this document. Most |
33 importantly, our guidelines should to make it easy for our tests to be | 33 importantly, our guidelines should to make it easy for our tests to be |
34 upstreamed. The `blink-dev` mailing list will be happy to help you harmonize our | 34 upstreamed. The |
35 current guidelines with communal test repositories. | 35 [blink-dev mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/
blink-dev) |
| 36 will be happy to help you harmonize our current guidelines with communal test |
| 37 repositories. |
36 *** | 38 *** |
37 | 39 |
38 ### Test Types | 40 ### Test Types |
39 | 41 |
40 There are four broad types of layout tests, listed in the order of preference. | 42 There are four broad types of layout tests, listed in the order of preference. |
41 | 43 |
42 * *JavaScript Tests* are the layout test implementation of | 44 * *JavaScript Tests* are the layout test implementation of |
43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain | 45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
44 assertions written in JavaScript, and pass if the assertions evaluate to | 46 assertions written in JavaScript, and pass if the assertions evaluate to |
45 true. | 47 true. |
(...skipping 20 matching lines...) Expand all Loading... |
66 match the baselines in the repository. DRT tests are less desirable than all | 68 match the baselines in the repository. DRT tests are less desirable than all |
67 the alternatives, because they depend on a browser implementation detail. | 69 the alternatives, because they depend on a browser implementation detail. |
68 | 70 |
69 ## General Principles | 71 ## General Principles |
70 | 72 |
71 The principles below are adapted from | 73 The principles below are adapted from |
72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) | 74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) |
73 and | 75 and |
74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). | 76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). |
75 | 77 |
76 * Tests should be **concise**, without compromising on the principles below. | |
77 Every element and piece of code on the page should be necessary and relevant | |
78 to what is being tested. For example, don't build a fully functional signup | |
79 form if you only need a text field or a button. | |
80 * Content needed to satisfy the principles below is considered necessary. | |
81 For example, it is acceptable and desirable to add elements that make | |
82 the test self-describing (see below), and to add code that makes the | |
83 test more reliable (see below). | |
84 * Content that makes test failures easier to debug is considered necessary | |
85 (to maintaining a good development speed), and is both acceptable and | |
86 desirable. | |
87 * Conciseness is particularly important for reference tests and pixel | |
88 tests, as the test pages are rendered in an 800x600px viewport. Having | |
89 content outside the viewport is undesirable because the outside content | |
90 does not get compared, and because the resulting scrollbars are | |
91 platform-specific UI widgets, making the test results less reliable. | |
92 | |
93 * Tests should be as **fast** as possible, without compromising on the | |
94 principles below. Blink has several thousand layout tests that are run in | |
95 parallel, and avoiding unnecessary delays is crucial to keeping our Commit | |
96 Queue in good shape. | |
97 * Avoid [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API
/WindowTimers/setTimeout), | |
98 as it wastes time on the testing infrastructure. Instead, use specific | |
99 event handlers, such as | |
100 [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEve
ntHandlers/onload), | |
101 to decide when to advance to the next step in a test. | |
102 | |
103 * Tests should be **reliable** and yield consistent results for a given | |
104 implementation. Flaky tests slow down your fellow developers' debugging | |
105 efforts and the Commit Queue. | |
106 * `window.setTimeout` is again a primary offender here. Asides from wasting | |
107 time on a fast system, tests that rely on fixed timeouts can fail when run | |
108 on systems that are slower than expected. | |
109 * Follow the guidelines in this | |
110 [PSA on writing reliable layout tests](https://docs.google.com/document/d/
1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit). | |
111 | |
112 * Tests should be **self-describing**, so that a project member can recognize | |
113 whether a test passes or fails without having to read the specification of the | |
114 feature being tested. `testharness.js` makes a test self-describing when used | |
115 correctly, but tests that degrade to manual tests | |
116 [must be carefully designed](http://testthewebforward.org/docs/test-style-guid
elines.html) | |
117 to be self-describing. | |
118 | |
119 * Tests should require a **minimal** amount of cognitive effort to read and | |
120 maintain. | |
121 * Avoid depending on edge case behavior of features that aren't explicitly | |
122 covered by the test. For example, except where testing parsing, tests | |
123 should contain valid markup (no parsing errors). | |
124 * Tests should provide as much relevant information as possible when | |
125 failing. `testharness.js` tests should prefer | |
126 [rich assert_ functions](https://github.com/w3c/testharness.js/blob/master
/docs/api.md#list-of-assertions) | |
127 to combining `assert_true()` with a boolean operator. Using appropriate | |
128 `assert_` functions results in better diagnostic output when the assertion | |
129 fails. | |
130 * Prefer JavaScript's | |
131 [===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operator
s/Comparison_Operators#Identity_strict_equality_()) | |
132 operator to | |
133 [==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators
/Comparison_Operators#Equality_()) | |
134 so that readers don't have to reason about | |
135 [type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abst
ract-equality-comparison). | |
136 | |
137 * Tests should be as **cross-platform** as reasonably possible. Avoid | |
138 assumptions about device type, screen resolution, etc. Unavoidable assumptions | |
139 should be documented. | |
140 * When possible, tests should only use Web platform features, as specified | |
141 in the relevant standards. When the Web platform's APIs are insufficient, | |
142 tests should prefer to use WPT extended testing APIs, such as | |
143 `wpt_automation`. | |
144 * Test pages should use the HTML5 doctype (`<!doctype html>`) unless they | |
145 specifically cover | |
146 [quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards
_Mode) | |
147 behavior. | |
148 * Tests should be written under the assumption that they will be upstreamed | |
149 to the WPT project. For example, tests should follow the | |
150 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). | |
151 * Tests that use Blink-specific testing APIs should feature-test for the | |
152 presence of the testing APIs and degrade to | |
153 [manual tests](http://testthewebforward.org/docs/manual-test.html) when | |
154 the testing APIs are not present. _This is not currently enforced in code | |
155 review. However, please keep in mind that a manual test can be debugged in | |
156 the browser, whereas a test that does not degrade gracefully can only be | |
157 debugged in the test runner._ | |
158 | |
159 * Tests must be **self-contained** and not depend on external network resources. | |
160 Unless used by multiple test files, CSS and JavaScript should be inlined using | |
161 `<style>` and `<script>` tags. Content shared by multiple tests should be | |
162 placed in a `resources/` directory near the tests that share it. See below for | |
163 using multiple origins in a test. | |
164 | |
165 * Test **file names** should describe what is being tested. File names should | |
166 use `snake-case`, but preserve the case of any embedded API names. For | |
167 example, prefer `document-createElement.html` to | |
168 `document-create-element.html`. | |
169 | |
170 * Tests should prefer **modern features** in JavaScript and in the Web Platform. | |
171 * Tests should use | |
172 [strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe
rence/Strict_mode) | |
173 for all JavaScript, except when specifically testing sloppy mode behavior. | |
174 Strict mode flags deprecated features and helps catch some errors, such as | |
175 forgetting to declare variables. | |
176 * JavaScript code should prefer | |
177 [const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statem
ents/const) | |
178 and | |
179 [let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statemen
ts/let) | |
180 over `var`, | |
181 [classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Clas
ses) | |
182 over other OOP constructs, and | |
183 [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Glo
bal_Objects/Promise) | |
184 over other mechanisms for structuring asynchronous code. | |
185 * The desire to use modern features must be balanced with the desire for | |
186 cross-platform tests. Avoid using features that haven't been shipped by | |
187 other developed major rendering engines (WebKit, Gecko, Edge). When | |
188 unsure, check [caniuse.com](http://caniuse.com/). | |
189 | |
190 * Tests should use the UTF-8 **character encoding**, which should be declared by | |
191 `<meta charset=utf-8>`. This does not apply when specifically testing | |
192 encodings. | |
193 * At this time, code reviewers may choose to accept layout tests that do | |
194 not have a `<meta charset>`, as long as the file content is pure ASCII. | |
195 If going that route, please keep in mind that Firefox currently issues a | |
196 dev tools warning for pages without a declared charset. | |
197 | |
198 * Tests should aim to have a **coding style** that is consistent with | |
199 [Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.
html), | |
200 and | |
201 [Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssgui
de.xml), | |
202 with the following exceptions. | |
203 * Rules related to Google Closure and JSDoc do not apply. | |
204 * Modern Web Platform and JavaScript features should be preferred to legacy | |
205 constructs that target old browsers. For example, prefer `const` and `let` | |
206 to `var`, and prefer `class` over other OOP constructs. This should be | |
207 balanced with the desire to have cross-platform tests. | |
208 * Concerns regarding buggy behavior in legacy browsers do not apply. For | |
209 example, the garbage collection cycle note in the _Closures_ section does | |
210 not apply. | |
211 * Per the JavaScript guide, new tests should also follow any per-project | |
212 style guide, such as the | |
213 [ServiceWorker Tests Style guide](http://www.chromium.org/blink/servicewor
ker/testing). | |
214 | |
215 *** note | 78 *** note |
216 This document intentionally uses _should_ a lot more than _must_, as defined in | 79 This document intentionally uses _should_ a lot more than _must_, as defined in |
217 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a | 80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a |
218 careful act of balancing many concerns, and this humble document cannot possibly | 81 careful act of balancing many concerns, and this humble document cannot possibly |
219 capture the context that rests in the head of an experienced Blink engineer. | 82 capture the context that rests in the head of an experienced Blink engineer. |
220 *** | 83 *** |
221 | 84 |
| 85 ### Concise |
| 86 |
| 87 Tests should be **concise**, without compromising on the principles below. Every |
| 88 element and piece of code on the page should be necessary and relevant to what |
| 89 is being tested. For example, don't build a fully functional signup form if you |
| 90 only need a text field or a button. |
| 91 |
| 92 Content needed to satisfy the principles below is considered necessary. For |
| 93 example, it is acceptable and desirable to add elements that make the test |
| 94 self-describing (see below), and to add code that makes the test more reliable |
| 95 (see below). |
| 96 |
| 97 Content that makes test failures easier to debug is considered necessary (to |
| 98 maintaining a good development speed), and is both acceptable and desirable. |
| 99 |
| 100 *** promo |
| 101 Conciseness is particularly important for reference tests and pixel tests, as |
| 102 the test pages are rendered in an 800x600px viewport. Having content outside the |
| 103 viewport is undesirable because the outside content does not get compared, and |
| 104 because the resulting scrollbars are platform-specific UI widgets, making the |
| 105 test results less reliable. |
| 106 *** |
| 107 |
| 108 ### Fast |
| 109 |
| 110 Tests should be as **fast** as possible, without compromising on the principles |
| 111 below. Blink has several thousand layout tests that are run in parallel, and |
| 112 avoiding unnecessary delays is crucial to keeping our Commit Queue in good |
| 113 shape. |
| 114 |
| 115 Avoid |
| 116 [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimer
s/setTimeout), |
| 117 as it wastes time on the testing infrastructure. Instead, use specific event |
| 118 handlers, such as |
| 119 [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHand
lers/onload), |
| 120 to decide when to advance to the next step in a test. |
| 121 |
| 122 ### Reliable |
| 123 |
| 124 Tests should be **reliable** and yield consistent results for a given |
| 125 implementation. Flaky tests slow down your fellow developers' debugging efforts |
| 126 and the Commit Queue. |
| 127 |
| 128 `window.setTimeout` is again a primary offender here. Asides from wasting time |
| 129 on a fast system, tests that rely on fixed timeouts can fail when on systems |
| 130 that are slower than expected. |
| 131 |
| 132 Follow the guidelines in this |
| 133 [PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4Sn
TLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit). |
| 134 |
| 135 ### Self-Describing |
| 136 |
| 137 Tests should be **self-describing**, so that a project member can recognize |
| 138 whether a test passes or fails without having to read the specification of the |
| 139 feature being tested. `testharness.js` makes a test self-describing when used |
| 140 correctly, but tests that degrade to manual tests |
| 141 [must be carefully designed](http://testthewebforward.org/docs/test-style-guidel
ines.html) |
| 142 to be self-describing. |
| 143 |
| 144 ### Minimal |
| 145 |
| 146 Tests should require a **minimal** amount of cognitive effort to read and |
| 147 maintain. |
| 148 |
| 149 Avoid depending on edge case behavior of features that aren't explicitly covered |
| 150 by the test. For example, except where testing parsing, tests should contain |
| 151 valid markup (no parsing errors). |
| 152 |
| 153 Tests should provide as much relevant information as possible when failing. |
| 154 `testharness.js` tests should prefer |
| 155 [rich assert_ functions](https://github.com/w3c/testharness.js/blob/master/docs/
api.md#list-of-assertions) |
| 156 to combining `assert_true()` with a boolean operator. Using appropriate |
| 157 `assert_` functions results in better diagnostic output when the assertion |
| 158 fails. |
| 159 |
| 160 🚧 Prefer JavaScript's |
| 161 [===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comp
arison_Operators#Identity_strict_equality_()) |
| 162 operator to |
| 163 [==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Compa
rison_Operators#Equality_()) |
| 164 so that readers don't have to reason about |
| 165 [type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abstract-e
quality-comparison). |
| 166 |
| 167 *** promo |
| 168 The === vs == recommendation is still being discussed on |
| 169 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 170 However, please keep in mind that a fellow developer who has to debug a failing |
| 171 test may have to consider the |
| 172 [special cases for ==](http://dorey.github.io/JavaScript-Equality-Table/) when |
| 173 the types of the two values being compared aren't immediately obvious. |
| 174 *** |
| 175 |
| 176 ### Cross-Platform |
| 177 |
| 178 Tests should be as **cross-platform** as reasonably possible. Avoid assumptions |
| 179 about device type, screen resolution, etc. Unavoidable assumptions should be |
| 180 documented. |
| 181 |
| 182 When possible, tests should only use Web platform features, as specified |
| 183 in the relevant standards. When the Web platform's APIs are insufficient, |
| 184 tests should prefer to use WPT extended testing APIs, such as |
| 185 `wpt_automation`, over Blink-specific testing APIs. |
| 186 |
| 187 🚧 Tests that use testing APIs should feature-test for the presence of |
| 188 those APIs, and gracefully degrade to manual tests (see below) when the testing |
| 189 APIs are not available. |
| 190 |
| 191 Test pages should use the HTML5 doctype (`<!doctype html>`) unless they |
| 192 specifically cover |
| 193 [quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards_Mode) |
| 194 behavior. |
| 195 |
| 196 Tests should be written under the assumption that they will be upstreamed |
| 197 to the WPT project. For example, tests should follow the |
| 198 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). |
| 199 |
| 200 Tests should avoid using features that haven't been shipped by the |
| 201 actively-developed major rendering engines (Blink, WebKit, Gecko, Edge). When |
| 202 unsure, check [caniuse.com](http://caniuse.com/). By necessity, this |
| 203 recommendation does not apply to the feature targeted by the test. |
| 204 |
| 205 *** note |
| 206 It may be tempting have a test for a bleeding-edge feature X depend on feature |
| 207 Y, which has only shipped in beta / development versions of various browsers. |
| 208 The reasoning would be that all browsers that implement X will have implemented |
| 209 Y. Please keep in mind that Chrome has un-shipped features that made it to the |
| 210 Beta channel in the past. |
| 211 *** |
| 212 |
| 213 Tests that use Blink-specific testing APIs should feature-test for the |
| 214 presence of the testing APIs and degrade to |
| 215 [manual tests](http://testthewebforward.org/docs/manual-test.html) when |
| 216 the testing APIs are not present. |
| 217 |
| 218 *** promo |
| 219 The recommendation to degrade to manual tests is still being discussed on |
| 220 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 221 However, please keep in mind that a manual test can be debugged in the browser, |
| 222 whereas a test that does not degrade gracefully can only be debugged in the test |
| 223 runner. Fellow project members and future you will thank you for having your |
| 224 test work as a manual test. |
| 225 *** |
| 226 |
| 227 ### Self-Contained |
| 228 |
| 229 Tests must be **self-contained** and not depend on external network resources. |
| 230 |
| 231 Unless used by multiple test files, CSS and JavaScript should be inlined using |
| 232 `<style>` and `<script>` tags. Content shared by multiple tests should be |
| 233 placed in a `resources/` directory near the tests that share it. See below for |
| 234 using multiple origins in a test. |
| 235 |
| 236 ### File Names |
| 237 |
| 238 Test **file names** should describe what is being tested. |
| 239 |
| 240 File names should use `snake-case`, but preserve the case of any embedded API |
| 241 names. For example, prefer `document-createElement.html` to |
| 242 `document-create-element.html`. |
| 243 |
| 244 ### Modern Features |
| 245 |
| 246 Tests should prefer **modern features** in JavaScript and in the Web Platform, |
| 247 provided that they meet the recommendations above for cross-platform tests. |
| 248 |
| 249 Tests should use |
| 250 [strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/
Strict_mode) |
| 251 for all JavaScript, except when specifically testing sloppy mode behavior. |
| 252 Strict mode flags deprecated features and helps catch some errors, such as |
| 253 forgetting to declare variables. |
| 254 |
| 255 🚧 JavaScript code should prefer |
| 256 [const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/c
onst) |
| 257 and |
| 258 [let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/let
) |
| 259 over `var`, |
| 260 [classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Classes) |
| 261 over other OOP constructs, and |
| 262 [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Ob
jects/Promise) |
| 263 over other mechanisms for structuring asynchronous code. |
| 264 |
| 265 *** promo |
| 266 The recommendation to prefer `const` and `let` over `var` is currently being |
| 267 discussed on |
| 268 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 269 *** |
| 270 |
| 271 ### Character Encoding |
| 272 |
| 273 Tests should use the UTF-8 **character encoding**, which should be declared by |
| 274 `<meta charset=utf-8>`. This does not apply when specifically testing encodings. |
| 275 |
| 276 The `<meta>` tag must be the first child of the document's `<head>` element. In |
| 277 documents that do not have an explicit `<head>`, the `<meta>` tag must follow |
| 278 the doctype. |
| 279 |
| 280 When HTML pages do not explicitly declare a character encoding, browsers |
| 281 determine the encoding using an |
| 282 [encoding sniffing algorithm](https://html.spec.whatwg.org/multipage/syntax.html
#determining-the-character-encoding) |
| 283 that will surprise most modern Web developers. Highlights include a default |
| 284 encoding that depends on the user's locale, and non-standardized |
| 285 browser-specific heuristics. |
| 286 |
| 287 *** promo |
| 288 The WPT guidelines state that test files that only contain ASCII characters may |
| 289 omit the `<meta>` tag. This exception is currently discussed on |
| 290 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 291 If taking that route, please keep in mind that Firefox currently issues a |
| 292 [development tools](https://developer.mozilla.org/en-US/docs/Tools) warning for |
| 293 pages without a declared encoding. |
| 294 *** |
| 295 |
| 296 ### Coding Style |
| 297 |
| 298 Tests should aim to have a **coding style** that is consistent with |
| 299 [Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.ht
ml), |
| 300 and |
| 301 [Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide
.xml), |
| 302 with the following exceptions. |
| 303 |
| 304 * Rules related to Google Closure and JSDoc do not apply. |
| 305 * Modern Web Platform and JavaScript features should be preferred to legacy |
| 306 constructs that target old browsers. |
| 307 * Per the JavaScript guide, new tests should also follow any per-project |
| 308 style guide, such as the |
| 309 [ServiceWorker Tests Style guide](http://www.chromium.org/blink/serviceworker/
testing). |
| 310 |
222 ## JavaScript Tests | 311 ## JavaScript Tests |
223 | 312 |
224 Whenever possible, the testing criteria should be expressed in JavaScript. The | 313 Whenever possible, the testing criteria should be expressed in JavaScript. The |
225 alternatives, which will be described in future sections, result in slower and | 314 alternatives, which will be described in future sections, result in slower and |
226 less reliable tests. | 315 less reliable tests. |
227 | 316 |
228 All new JavaScript tests should be written using the | 317 All new JavaScript tests should be written using the |
229 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This | 318 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This |
230 framework is used by the tests in the | 319 framework is used by the tests in the |
231 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, | 320 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, |
(...skipping 158 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
390 for other useful APIs. For example, `window.eventSender` | 479 for other useful APIs. For example, `window.eventSender` |
391 ([components/test_runner/event_sender.h](../../components/test_runner/event_send
er.h) | 480 ([components/test_runner/event_sender.h](../../components/test_runner/event_send
er.h) |
392 and | 481 and |
393 [components/test_runner/event_sender.cpp](../../components/test_runner/event_sen
der.cpp)) | 482 [components/test_runner/event_sender.cpp](../../components/test_runner/event_sen
der.cpp)) |
394 has methods that simulate events input such as keyboard / mouse input and | 483 has methods that simulate events input such as keyboard / mouse input and |
395 drag-and-drop. | 484 drag-and-drop. |
396 | 485 |
397 Here is a UML diagram of how the `testRunner` bindings fit into Chromium. | 486 Here is a UML diagram of how the `testRunner` bindings fit into Chromium. |
398 | 487 |
399 [](https://docs.google
.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) | 488 [](https://docs.google
.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) |
| 489 |
400 ### Manual Tests | 490 ### Manual Tests |
401 | 491 |
402 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs should | 492 🚧 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs |
403 also be usable as | 493 should also be usable as |
404 [manual tests](http://testthewebforward.org/docs/manual-test.html). This makes | 494 [manual tests](http://testthewebforward.org/docs/manual-test.html). This makes |
405 it easy to debug the test, and to check whether our behavior matches other | 495 it easy to debug the test, and to check whether our behavior matches other |
406 browsers. | 496 browsers. |
407 | 497 |
408 *** note | 498 *** promo |
409 The recommendation to have tests that depend on Blink-only testing APIs | 499 The recommendation to degrade to manual tests is still being discussed on |
410 gracefully degrade to manual tests is not currently enforced in code review. | 500 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
411 When considering skipping this recommendation, please keep in mind that a manual | 501 However, please keep in mind that a manual test can be debugged in the browser, |
412 test can be debugged in the browser, whereas a test that does not degrade | 502 whereas a test that does not degrade gracefully can only be debugged in the test |
413 gracefully can only be debugged in the test runner. Fellow project members and | 503 runner. Fellow project members and future you will thank you for having your |
414 future you will thank you for having your test work as a manual test. | 504 test work as a manual test. |
415 *** | 505 *** |
416 | 506 |
417 Manual tests should minimize the chance of user error. This implies keeping the | 507 Manual tests should minimize the chance of user error. This implies keeping the |
418 manual steps to a minimum, and having simple and clear instructions that | 508 manual steps to a minimum, and having simple and clear instructions that |
419 describe all the configuration changes and user gestures that match the effect | 509 describe all the configuration changes and user gestures that match the effect |
420 of the Blink-specific APIs used by the test. | 510 of the Blink-specific APIs used by the test. |
421 | 511 |
422 Below is an example of a fairly minimal test that uses a Blink-Specific API | 512 Below is an example of a fairly minimal test that uses a Blink-Specific API |
423 (`window.eventSender`), and gracefully degrades to a manual test. | 513 (`window.eventSender`), and gracefully degrades to a manual test. |
424 | 514 |
(...skipping 151 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
576 where it is useful to have a test pass when the two images do _not_ match. | 666 where it is useful to have a test pass when the two images do _not_ match. |
577 | 667 |
578 Reference tests are more difficult to debug than JavaScript tests, and tend to | 668 Reference tests are more difficult to debug than JavaScript tests, and tend to |
579 be slower as well. Therefore, they should only be used for functionality that | 669 be slower as well. Therefore, they should only be used for functionality that |
580 cannot be covered by JavaScript tests. | 670 cannot be covered by JavaScript tests. |
581 | 671 |
582 New reference tests should follow the | 672 New reference tests should follow the |
583 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The | 673 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The |
584 most important points are summarized below. | 674 most important points are summarized below. |
585 | 675 |
586 * The test page declares the reference page using a `<link rel="match">` or | 676 * 🚧 The test page declares the reference page using a |
587 `<link rel="mismatch">`, depending on whether the test passes when the test | 677 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test |
588 image matches or does not match the reference image. | 678 passes when the test image matches or does not match the reference image. |
589 * The reference page must not use the feature being tested. Otherwise, the test | 679 * The reference page must not use the feature being tested. Otherwise, the test |
590 is meaningless. | 680 is meaningless. |
591 * The reference page should be as simple as possible, and should not depend on | 681 * The reference page should be as simple as possible, and should not depend on |
592 advanced features. Ideally, the reference page should render as intended even | 682 advanced features. Ideally, the reference page should render as intended even |
593 on browsers with poor CSS support. | 683 on browsers with poor CSS support. |
594 * Reference tests should be self-describing. | 684 * Reference tests should be self-describing. |
595 * Reference tests do _not_ include `testharness.js`. | 685 * Reference tests do _not_ include `testharness.js`. |
596 | 686 |
597 Our testing infrastructure was designed for the | 687 🚧 Our testing infrastructure was designed for the |
598 [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink | 688 [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink |
599 has inherited. The consequences are summarized below. | 689 has inherited. The consequences are summarized below. |
600 | 690 |
601 * Each reference page must be in the same directory as its associated test. | 691 * Each reference page must be in the same directory as its associated test. |
602 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`), | 692 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`), |
603 * The reference page must be named `foo-expected` (e.g., | 693 * The reference page must be named `foo-expected` (e.g., |
604 `foo-expected.html`) if the test passes when the two images match. | 694 `foo-expected.html`) if the test passes when the two images match. |
605 * The reference page must be named `foo-expected-mismatch` (e.g., | 695 * The reference page must be named `foo-expected-mismatch` (e.g., |
606 `foo-expected-mismatch.svg`) if the test passes when the two images do | 696 `foo-expected-mismatch.svg`) if the test passes when the two images do |
607 _not_ match. | 697 _not_ match. |
(...skipping 21 matching lines...) Expand all Loading... |
629 <!doctype html> | 719 <!doctype html> |
630 <meta charset="utf-8"> | 720 <meta charset="utf-8"> |
631 | 721 |
632 <ol> | 722 <ol> |
633 <li value="3">A</li> | 723 <li value="3">A</li> |
634 <li value="2">B</li> | 724 <li value="2">B</li> |
635 <li value="1">C</li> | 725 <li value="1">C</li> |
636 </ol> | 726 </ol> |
637 ``` | 727 ``` |
638 | 728 |
| 729 *** promo |
| 730 The method for pointing out a test's reference page is still in flux, and is |
| 731 being discussed on |
| 732 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 733 *** |
| 734 |
639 ## Pixel Tests | 735 ## Pixel Tests |
640 | 736 |
641 `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and | 737 `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and |
642 `window.testRunner.dumpDragImage()` create an image result that is associated | 738 `window.testRunner.dumpDragImage()` create an image result that is associated |
643 with the test. The image result is compared against an image baseline, which is | 739 with the test. The image result is compared against an image baseline, which is |
644 an `-expected.png` file associated with the test, and the test passes if the | 740 an `-expected.png` file associated with the test, and the test passes if the |
645 image result is identical to the baseline, according to a pixel-by-pixel | 741 image result is identical to the baseline, according to a pixel-by-pixel |
646 comparison. Tests that have image results (and baselines) are called **pixel | 742 comparison. Tests that have image results (and baselines) are called **pixel |
647 tests**. | 743 tests**. |
648 | 744 |
649 Pixel tests should still follow the principles laid out above. Pixel tests pose | 745 Pixel tests should still follow the principles laid out above. Pixel tests pose |
650 unique challenges to the desire to have *self-describing* and *cross-platform* | 746 unique challenges to the desire to have *self-describing* and *cross-platform* |
651 tests. The | 747 tests. The |
652 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli
nes.html) | 748 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli
nes.html) |
653 contain useful guidance. The most relevant pieces of advice are below. | 749 contain useful guidance. The most relevant pieces of advice are below. |
654 | 750 |
655 * Whenever possible, use a green paragraph / page / square to indicate success. | 751 * Whenever possible, use a green paragraph / page / square to indicate success. |
656 If that is not possible, make the test self-describing by including a textual | 752 If that is not possible, make the test self-describing by including a textual |
657 description of the desired (passing) outcome. | 753 description of the desired (passing) outcome. |
658 * Only use the red color or the word `FAIL` to highlight errors. This does not | 754 * Only use the red color or the word `FAIL` to highlight errors. This does not |
659 apply when testing the color red. | 755 apply when testing the color red. |
660 * Use the [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to | 756 * 🚧 Use the |
661 reduce the variance introduced by the platform's text rendering system. This | 757 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the |
662 does not apply when testing text, text flow, font selection, font fallback, | 758 variance introduced by the platform's text rendering system. This does not |
663 font features, or other typographic information. | 759 apply when testing text, text flow, font selection, font fallback, font |
| 760 features, or other typographic information. |
664 | 761 |
665 *** promo | 762 *** promo |
666 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result | 763 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result |
667 will always be 800x600px, because test pages are rendered in an 800x600px | 764 will always be 800x600px, because test pages are rendered in an 800x600px |
668 viewport. Pixel tests that do not specifically cover scrolling should fit in an | 765 viewport. Pixel tests that do not specifically cover scrolling should fit in an |
669 800x600px viewport without creating scrollbars. | 766 800x600px viewport without creating scrollbars. |
670 *** | 767 *** |
671 | 768 |
| 769 *** promo |
| 770 The recommendation of using Ahem in pixel tests is being discussed on |
| 771 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 772 *** |
| 773 |
672 The following snippet includes the Ahem font in a layout test. | 774 The following snippet includes the Ahem font in a layout test. |
673 | 775 |
674 ```html | 776 ```html |
675 <style> | 777 <style> |
676 body { | 778 body { |
677 font: 10px Ahem; | 779 font: 10px Ahem; |
678 } | 780 } |
679 </style> | 781 </style> |
680 <script src="/resources/ahem.js"></script> | 782 <script src="/resources/ahem.js"></script> |
681 ``` | 783 ``` |
(...skipping 96 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
778 * The `http/` directory hosts tests that require an HTTP server (see above). | 880 * The `http/` directory hosts tests that require an HTTP server (see above). |
779 * The `resources/` subdirectory in every directory contains binary files, such | 881 * The `resources/` subdirectory in every directory contains binary files, such |
780 as media files, and code that is shared by multiple test files. | 882 as media files, and code that is shared by multiple test files. |
781 | 883 |
782 *** note | 884 *** note |
783 Some layout tests consist of a minimal HTML page that references a JavaScript | 885 Some layout tests consist of a minimal HTML page that references a JavaScript |
784 file in `resources/`. Please do not use this pattern for new tests, as it goes | 886 file in `resources/`. Please do not use this pattern for new tests, as it goes |
785 against the minimality principle. JavaScript and CSS files should only live in | 887 against the minimality principle. JavaScript and CSS files should only live in |
786 `resources/` if they are shared by at least two test files. | 888 `resources/` if they are shared by at least two test files. |
787 *** | 889 *** |
OLD | NEW |