| OLD | NEW |
| 1 # Writing Layout Tests | 1 # Writing Layout Tests |
| 2 | 2 |
| 3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
| 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), |
| 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
| 6 or XHTML) and lives in | 6 or XHTML) and lives in |
| 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
| 8 | 8 |
| 9 [TOC] | 9 [TOC] |
| 10 | 10 |
| (...skipping 13 matching lines...) Expand all Loading... |
| 24 cannot be easily covered by | 24 cannot be easily covered by |
| 25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour
ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), | 25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour
ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), |
| 26 the feature must be covered by layout tests, to avoid unexpected regressions. | 26 the feature must be covered by layout tests, to avoid unexpected regressions. |
| 27 These tests will use Blink-specific testing APIs that are only available in | 27 These tests will use Blink-specific testing APIs that are only available in |
| 28 [content_shell](./layout_tests_in_content_shell.md). | 28 [content_shell](./layout_tests_in_content_shell.md). |
| 29 | 29 |
| 30 *** promo | 30 *** promo |
| 31 If you know that Blink layout tests are upstreamed to other projects, such as | 31 If you know that Blink layout tests are upstreamed to other projects, such as |
| 32 [test262](https://github.com/tc39/test262), please update this document. Most | 32 [test262](https://github.com/tc39/test262), please update this document. Most |
| 33 importantly, our guidelines should to make it easy for our tests to be | 33 importantly, our guidelines should to make it easy for our tests to be |
| 34 upstreamed. The `blink-dev` mailing list will be happy to help you harmonize our | 34 upstreamed. The |
| 35 current guidelines with communal test repositories. | 35 [blink-dev mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/
blink-dev) |
| 36 will be happy to help you harmonize our current guidelines with communal test |
| 37 repositories. |
| 36 *** | 38 *** |
| 37 | 39 |
| 38 ### Test Types | 40 ### Test Types |
| 39 | 41 |
| 40 There are four broad types of layout tests, listed in the order of preference. | 42 There are four broad types of layout tests, listed in the order of preference. |
| 41 | 43 |
| 42 * *JavaScript Tests* are the layout test implementation of | 44 * *JavaScript Tests* are the layout test implementation of |
| 43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain | 45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
| 44 assertions written in JavaScript, and pass if the assertions evaluate to | 46 assertions written in JavaScript, and pass if the assertions evaluate to |
| 45 true. | 47 true. |
| (...skipping 20 matching lines...) Expand all Loading... |
| 66 match the baselines in the repository. DRT tests are less desirable than all | 68 match the baselines in the repository. DRT tests are less desirable than all |
| 67 the alternatives, because they depend on a browser implementation detail. | 69 the alternatives, because they depend on a browser implementation detail. |
| 68 | 70 |
| 69 ## General Principles | 71 ## General Principles |
| 70 | 72 |
| 71 The principles below are adapted from | 73 The principles below are adapted from |
| 72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) | 74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) |
| 73 and | 75 and |
| 74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). | 76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). |
| 75 | 77 |
| 76 * Tests should be **concise**, without compromising on the principles below. | |
| 77 Every element and piece of code on the page should be necessary and relevant | |
| 78 to what is being tested. For example, don't build a fully functional signup | |
| 79 form if you only need a text field or a button. | |
| 80 * Content needed to satisfy the principles below is considered necessary. | |
| 81 For example, it is acceptable and desirable to add elements that make | |
| 82 the test self-describing (see below), and to add code that makes the | |
| 83 test more reliable (see below). | |
| 84 * Content that makes test failures easier to debug is considered necessary | |
| 85 (to maintaining a good development speed), and is both acceptable and | |
| 86 desirable. | |
| 87 * Conciseness is particularly important for reference tests and pixel | |
| 88 tests, as the test pages are rendered in an 800x600px viewport. Having | |
| 89 content outside the viewport is undesirable because the outside content | |
| 90 does not get compared, and because the resulting scrollbars are | |
| 91 platform-specific UI widgets, making the test results less reliable. | |
| 92 | |
| 93 * Tests should be as **fast** as possible, without compromising on the | |
| 94 principles below. Blink has several thousand layout tests that are run in | |
| 95 parallel, and avoiding unnecessary delays is crucial to keeping our Commit | |
| 96 Queue in good shape. | |
| 97 * Avoid [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API
/WindowTimers/setTimeout), | |
| 98 as it wastes time on the testing infrastructure. Instead, use specific | |
| 99 event handlers, such as | |
| 100 [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEve
ntHandlers/onload), | |
| 101 to decide when to advance to the next step in a test. | |
| 102 | |
| 103 * Tests should be **reliable** and yield consistent results for a given | |
| 104 implementation. Flaky tests slow down your fellow developers' debugging | |
| 105 efforts and the Commit Queue. | |
| 106 * `window.setTimeout` is again a primary offender here. Asides from wasting | |
| 107 time on a fast system, tests that rely on fixed timeouts can fail when run | |
| 108 on systems that are slower than expected. | |
| 109 * Follow the guidelines in this | |
| 110 [PSA on writing reliable layout tests](https://docs.google.com/document/d/
1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit). | |
| 111 | |
| 112 * Tests should be **self-describing**, so that a project member can recognize | |
| 113 whether a test passes or fails without having to read the specification of the | |
| 114 feature being tested. `testharness.js` makes a test self-describing when used | |
| 115 correctly, but tests that degrade to manual tests | |
| 116 [must be carefully designed](http://testthewebforward.org/docs/test-style-guid
elines.html) | |
| 117 to be self-describing. | |
| 118 | |
| 119 * Tests should require a **minimal** amount of cognitive effort to read and | |
| 120 maintain. | |
| 121 * Avoid depending on edge case behavior of features that aren't explicitly | |
| 122 covered by the test. For example, except where testing parsing, tests | |
| 123 should contain valid markup (no parsing errors). | |
| 124 * Tests should provide as much relevant information as possible when | |
| 125 failing. `testharness.js` tests should prefer | |
| 126 [rich assert_ functions](https://github.com/w3c/testharness.js/blob/master
/docs/api.md#list-of-assertions) | |
| 127 to combining `assert_true()` with a boolean operator. Using appropriate | |
| 128 `assert_` functions results in better diagnostic output when the assertion | |
| 129 fails. | |
| 130 * Prefer JavaScript's | |
| 131 [===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operator
s/Comparison_Operators#Identity_strict_equality_()) | |
| 132 operator to | |
| 133 [==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators
/Comparison_Operators#Equality_()) | |
| 134 so that readers don't have to reason about | |
| 135 [type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abst
ract-equality-comparison). | |
| 136 | |
| 137 * Tests should be as **cross-platform** as reasonably possible. Avoid | |
| 138 assumptions about device type, screen resolution, etc. Unavoidable assumptions | |
| 139 should be documented. | |
| 140 * When possible, tests should only use Web platform features, as specified | |
| 141 in the relevant standards. When the Web platform's APIs are insufficient, | |
| 142 tests should prefer to use WPT extended testing APIs, such as | |
| 143 `wpt_automation`. | |
| 144 * Test pages should use the HTML5 doctype (`<!doctype html>`) unless they | |
| 145 specifically cover | |
| 146 [quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards
_Mode) | |
| 147 behavior. | |
| 148 * Tests should be written under the assumption that they will be upstreamed | |
| 149 to the WPT project. For example, tests should follow the | |
| 150 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). | |
| 151 * Tests that use Blink-specific testing APIs should feature-test for the | |
| 152 presence of the testing APIs and degrade to | |
| 153 [manual tests](http://testthewebforward.org/docs/manual-test.html) when | |
| 154 the testing APIs are not present. _This is not currently enforced in code | |
| 155 review. However, please keep in mind that a manual test can be debugged in | |
| 156 the browser, whereas a test that does not degrade gracefully can only be | |
| 157 debugged in the test runner._ | |
| 158 | |
| 159 * Tests must be **self-contained** and not depend on external network resources. | |
| 160 Unless used by multiple test files, CSS and JavaScript should be inlined using | |
| 161 `<style>` and `<script>` tags. Content shared by multiple tests should be | |
| 162 placed in a `resources/` directory near the tests that share it. See below for | |
| 163 using multiple origins in a test. | |
| 164 | |
| 165 * Test **file names** should describe what is being tested. File names should | |
| 166 use `snake-case`, but preserve the case of any embedded API names. For | |
| 167 example, prefer `document-createElement.html` to | |
| 168 `document-create-element.html`. | |
| 169 | |
| 170 * Tests should prefer **modern features** in JavaScript and in the Web Platform. | |
| 171 * Tests should use | |
| 172 [strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe
rence/Strict_mode) | |
| 173 for all JavaScript, except when specifically testing sloppy mode behavior. | |
| 174 Strict mode flags deprecated features and helps catch some errors, such as | |
| 175 forgetting to declare variables. | |
| 176 * JavaScript code should prefer | |
| 177 [const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statem
ents/const) | |
| 178 and | |
| 179 [let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statemen
ts/let) | |
| 180 over `var`, | |
| 181 [classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Clas
ses) | |
| 182 over other OOP constructs, and | |
| 183 [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Glo
bal_Objects/Promise) | |
| 184 over other mechanisms for structuring asynchronous code. | |
| 185 * The desire to use modern features must be balanced with the desire for | |
| 186 cross-platform tests. Avoid using features that haven't been shipped by | |
| 187 other developed major rendering engines (WebKit, Gecko, Edge). When | |
| 188 unsure, check [caniuse.com](http://caniuse.com/). | |
| 189 | |
| 190 * Tests should use the UTF-8 **character encoding**, which should be declared by | |
| 191 `<meta charset=utf-8>`. This does not apply when specifically testing | |
| 192 encodings. | |
| 193 * At this time, code reviewers may choose to accept layout tests that do | |
| 194 not have a `<meta charset>`, as long as the file content is pure ASCII. | |
| 195 If going that route, please keep in mind that Firefox currently issues a | |
| 196 dev tools warning for pages without a declared charset. | |
| 197 | |
| 198 * Tests should aim to have a **coding style** that is consistent with | |
| 199 [Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.
html), | |
| 200 and | |
| 201 [Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssgui
de.xml), | |
| 202 with the following exceptions. | |
| 203 * Rules related to Google Closure and JSDoc do not apply. | |
| 204 * Modern Web Platform and JavaScript features should be preferred to legacy | |
| 205 constructs that target old browsers. For example, prefer `const` and `let` | |
| 206 to `var`, and prefer `class` over other OOP constructs. This should be | |
| 207 balanced with the desire to have cross-platform tests. | |
| 208 * Concerns regarding buggy behavior in legacy browsers do not apply. For | |
| 209 example, the garbage collection cycle note in the _Closures_ section does | |
| 210 not apply. | |
| 211 * Per the JavaScript guide, new tests should also follow any per-project | |
| 212 style guide, such as the | |
| 213 [ServiceWorker Tests Style guide](http://www.chromium.org/blink/servicewor
ker/testing). | |
| 214 | |
| 215 *** note | 78 *** note |
| 216 This document intentionally uses _should_ a lot more than _must_, as defined in | 79 This document intentionally uses _should_ a lot more than _must_, as defined in |
| 217 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a | 80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a |
| 218 careful act of balancing many concerns, and this humble document cannot possibly | 81 careful act of balancing many concerns, and this humble document cannot possibly |
| 219 capture the context that rests in the head of an experienced Blink engineer. | 82 capture the context that rests in the head of an experienced Blink engineer. |
| 220 *** | 83 *** |
| 221 | 84 |
| 85 ### Concise |
| 86 |
| 87 Tests should be **concise**, without compromising on the principles below. Every |
| 88 element and piece of code on the page should be necessary and relevant to what |
| 89 is being tested. For example, don't build a fully functional signup form if you |
| 90 only need a text field or a button. |
| 91 |
| 92 Content needed to satisfy the principles below is considered necessary. For |
| 93 example, it is acceptable and desirable to add elements that make the test |
| 94 self-describing (see below), and to add code that makes the test more reliable |
| 95 (see below). |
| 96 |
| 97 Content that makes test failures easier to debug is considered necessary (to |
| 98 maintaining a good development speed), and is both acceptable and desirable. |
| 99 |
| 100 *** promo |
| 101 Conciseness is particularly important for reference tests and pixel tests, as |
| 102 the test pages are rendered in an 800x600px viewport. Having content outside the |
| 103 viewport is undesirable because the outside content does not get compared, and |
| 104 because the resulting scrollbars are platform-specific UI widgets, making the |
| 105 test results less reliable. |
| 106 *** |
| 107 |
| 108 ### Fast |
| 109 |
| 110 Tests should be as **fast** as possible, without compromising on the principles |
| 111 below. Blink has several thousand layout tests that are run in parallel, and |
| 112 avoiding unnecessary delays is crucial to keeping our Commit Queue in good |
| 113 shape. |
| 114 |
| 115 Avoid |
| 116 [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimer
s/setTimeout), |
| 117 as it wastes time on the testing infrastructure. Instead, use specific event |
| 118 handlers, such as |
| 119 [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHand
lers/onload), |
| 120 to decide when to advance to the next step in a test. |
| 121 |
| 122 ### Reliable |
| 123 |
| 124 Tests should be **reliable** and yield consistent results for a given |
| 125 implementation. Flaky tests slow down your fellow developers' debugging efforts |
| 126 and the Commit Queue. |
| 127 |
| 128 `window.setTimeout` is again a primary offender here. Asides from wasting time |
| 129 on a fast system, tests that rely on fixed timeouts can fail when on systems |
| 130 that are slower than expected. |
| 131 |
| 132 Follow the guidelines in this |
| 133 [PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4Sn
TLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit). |
| 134 |
| 135 ### Self-Describing |
| 136 |
| 137 Tests should be **self-describing**, so that a project member can recognize |
| 138 whether a test passes or fails without having to read the specification of the |
| 139 feature being tested. `testharness.js` makes a test self-describing when used |
| 140 correctly, but tests that degrade to manual tests |
| 141 [must be carefully designed](http://testthewebforward.org/docs/test-style-guidel
ines.html) |
| 142 to be self-describing. |
| 143 |
| 144 ### Minimal |
| 145 |
| 146 Tests should require a **minimal** amount of cognitive effort to read and |
| 147 maintain. |
| 148 |
| 149 Avoid depending on edge case behavior of features that aren't explicitly covered |
| 150 by the test. For example, except where testing parsing, tests should contain |
| 151 valid markup (no parsing errors). |
| 152 |
| 153 Tests should provide as much relevant information as possible when failing. |
| 154 `testharness.js` tests should prefer |
| 155 [rich assert_ functions](https://github.com/w3c/testharness.js/blob/master/docs/
api.md#list-of-assertions) |
| 156 to combining `assert_true()` with a boolean operator. Using appropriate |
| 157 `assert_` functions results in better diagnostic output when the assertion |
| 158 fails. |
| 159 |
| 160 🚧 Prefer JavaScript's |
| 161 [===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comp
arison_Operators#Identity_strict_equality_()) |
| 162 operator to |
| 163 [==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Compa
rison_Operators#Equality_()) |
| 164 so that readers don't have to reason about |
| 165 [type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abstract-e
quality-comparison). |
| 166 |
| 167 *** promo |
| 168 The === vs == recommendation is still being discussed on |
| 169 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 170 However, please keep in mind that a fellow developer who has to debug a failing |
| 171 test may have to consider the |
| 172 [special cases for ==](http://dorey.github.io/JavaScript-Equality-Table/) when |
| 173 the types of the two values being compared aren't immediately obvious. |
| 174 *** |
| 175 |
| 176 ### Cross-Platform |
| 177 |
| 178 Tests should be as **cross-platform** as reasonably possible. Avoid assumptions |
| 179 about device type, screen resolution, etc. Unavoidable assumptions should be |
| 180 documented. |
| 181 |
| 182 When possible, tests should only use Web platform features, as specified |
| 183 in the relevant standards. When the Web platform's APIs are insufficient, |
| 184 tests should prefer to use WPT extended testing APIs, such as |
| 185 `wpt_automation`, over Blink-specific testing APIs. |
| 186 |
| 187 🚧 Tests that use testing APIs should feature-test for the presence of |
| 188 those APIs, and gracefully degrade to manual tests (see below) when the testing |
| 189 APIs are not available. |
| 190 |
| 191 Test pages should use the HTML5 doctype (`<!doctype html>`) unless they |
| 192 specifically cover |
| 193 [quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards_Mode) |
| 194 behavior. |
| 195 |
| 196 Tests should be written under the assumption that they will be upstreamed |
| 197 to the WPT project. For example, tests should follow the |
| 198 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). |
| 199 |
| 200 Tests should avoid using features that haven't been shipped by the |
| 201 actively-developed major rendering engines (Blink, WebKit, Gecko, Edge). When |
| 202 unsure, check [caniuse.com](http://caniuse.com/). By necessity, this |
| 203 recommendation does not apply to the feature targeted by the test. |
| 204 |
| 205 *** note |
| 206 It may be tempting have a test for a bleeding-edge feature X depend on feature |
| 207 Y, which has only shipped in beta / development versions of various browsers. |
| 208 The reasoning would be that all browsers that implement X will have implemented |
| 209 Y. Please keep in mind that Chrome has un-shipped features that made it to the |
| 210 Beta channel in the past. |
| 211 *** |
| 212 |
| 213 Tests that use Blink-specific testing APIs should feature-test for the |
| 214 presence of the testing APIs and degrade to |
| 215 [manual tests](http://testthewebforward.org/docs/manual-test.html) when |
| 216 the testing APIs are not present. |
| 217 |
| 218 *** promo |
| 219 The recommendation to degrade to manual tests is still being discussed on |
| 220 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 221 However, please keep in mind that a manual test can be debugged in the browser, |
| 222 whereas a test that does not degrade gracefully can only be debugged in the test |
| 223 runner. Fellow project members and future you will thank you for having your |
| 224 test work as a manual test. |
| 225 *** |
| 226 |
| 227 ### Self-Contained |
| 228 |
| 229 Tests must be **self-contained** and not depend on external network resources. |
| 230 |
| 231 Unless used by multiple test files, CSS and JavaScript should be inlined using |
| 232 `<style>` and `<script>` tags. Content shared by multiple tests should be |
| 233 placed in a `resources/` directory near the tests that share it. See below for |
| 234 using multiple origins in a test. |
| 235 |
| 236 ### File Names |
| 237 |
| 238 Test **file names** should describe what is being tested. |
| 239 |
| 240 File names should use `snake-case`, but preserve the case of any embedded API |
| 241 names. For example, prefer `document-createElement.html` to |
| 242 `document-create-element.html`. |
| 243 |
| 244 ### Modern Features |
| 245 |
| 246 Tests should prefer **modern features** in JavaScript and in the Web Platform, |
| 247 provided that they meet the recommendations above for cross-platform tests. |
| 248 |
| 249 Tests should use |
| 250 [strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/
Strict_mode) |
| 251 for all JavaScript, except when specifically testing sloppy mode behavior. |
| 252 Strict mode flags deprecated features and helps catch some errors, such as |
| 253 forgetting to declare variables. |
| 254 |
| 255 🚧 JavaScript code should prefer |
| 256 [const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/c
onst) |
| 257 and |
| 258 [let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/let
) |
| 259 over `var`, |
| 260 [classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Classes) |
| 261 over other OOP constructs, and |
| 262 [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Ob
jects/Promise) |
| 263 over other mechanisms for structuring asynchronous code. |
| 264 |
| 265 *** promo |
| 266 The recommendation to prefer `const` and `let` over `var` is currently being |
| 267 discussed on |
| 268 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 269 *** |
| 270 |
| 271 ### Character Encoding |
| 272 |
| 273 Tests should use the UTF-8 **character encoding**, which should be declared by |
| 274 `<meta charset=utf-8>`. This does not apply when specifically testing encodings. |
| 275 |
| 276 The `<meta>` tag must be the first child of the document's `<head>` element. In |
| 277 documents that do not have an explicit `<head>`, the `<meta>` tag must follow |
| 278 the doctype. |
| 279 |
| 280 When HTML pages do not explicitly declare a character encoding, browsers |
| 281 determine the encoding using an |
| 282 [encoding sniffing algorithm](https://html.spec.whatwg.org/multipage/syntax.html
#determining-the-character-encoding) |
| 283 that will surprise most modern Web developers. Highlights include a default |
| 284 encoding that depends on the user's locale, and non-standardized |
| 285 browser-specific heuristics. |
| 286 |
| 287 *** promo |
| 288 The WPT guidelines state that test files that only contain ASCII characters may |
| 289 omit the `<meta>` tag. This exception is currently discussed on |
| 290 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 291 If taking that route, please keep in mind that Firefox currently issues a |
| 292 [development tools](https://developer.mozilla.org/en-US/docs/Tools) warning for |
| 293 pages without a declared encoding. |
| 294 *** |
| 295 |
| 296 ### Coding Style |
| 297 |
| 298 Tests should aim to have a **coding style** that is consistent with |
| 299 [Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.ht
ml), |
| 300 and |
| 301 [Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide
.xml), |
| 302 with the following exceptions. |
| 303 |
| 304 * Rules related to Google Closure and JSDoc do not apply. |
| 305 * Modern Web Platform and JavaScript features should be preferred to legacy |
| 306 constructs that target old browsers. |
| 307 * Per the JavaScript guide, new tests should also follow any per-project |
| 308 style guide, such as the |
| 309 [ServiceWorker Tests Style guide](http://www.chromium.org/blink/serviceworker/
testing). |
| 310 |
| 222 ## JavaScript Tests | 311 ## JavaScript Tests |
| 223 | 312 |
| 224 Whenever possible, the testing criteria should be expressed in JavaScript. The | 313 Whenever possible, the testing criteria should be expressed in JavaScript. The |
| 225 alternatives, which will be described in future sections, result in slower and | 314 alternatives, which will be described in future sections, result in slower and |
| 226 less reliable tests. | 315 less reliable tests. |
| 227 | 316 |
| 228 All new JavaScript tests should be written using the | 317 All new JavaScript tests should be written using the |
| 229 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This | 318 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This |
| 230 framework is used by the tests in the | 319 framework is used by the tests in the |
| 231 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, | 320 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, |
| (...skipping 158 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 390 for other useful APIs. For example, `window.eventSender` | 479 for other useful APIs. For example, `window.eventSender` |
| 391 ([components/test_runner/event_sender.h](../../components/test_runner/event_send
er.h) | 480 ([components/test_runner/event_sender.h](../../components/test_runner/event_send
er.h) |
| 392 and | 481 and |
| 393 [components/test_runner/event_sender.cpp](../../components/test_runner/event_sen
der.cpp)) | 482 [components/test_runner/event_sender.cpp](../../components/test_runner/event_sen
der.cpp)) |
| 394 has methods that simulate events input such as keyboard / mouse input and | 483 has methods that simulate events input such as keyboard / mouse input and |
| 395 drag-and-drop. | 484 drag-and-drop. |
| 396 | 485 |
| 397 Here is a UML diagram of how the `testRunner` bindings fit into Chromium. | 486 Here is a UML diagram of how the `testRunner` bindings fit into Chromium. |
| 398 | 487 |
| 399 [](https://docs.google
.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) | 488 [](https://docs.google
.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) |
| 489 |
| 400 ### Manual Tests | 490 ### Manual Tests |
| 401 | 491 |
| 402 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs should | 492 🚧 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs |
| 403 also be usable as | 493 should also be usable as |
| 404 [manual tests](http://testthewebforward.org/docs/manual-test.html). This makes | 494 [manual tests](http://testthewebforward.org/docs/manual-test.html). This makes |
| 405 it easy to debug the test, and to check whether our behavior matches other | 495 it easy to debug the test, and to check whether our behavior matches other |
| 406 browsers. | 496 browsers. |
| 407 | 497 |
| 408 *** note | 498 *** promo |
| 409 The recommendation to have tests that depend on Blink-only testing APIs | 499 The recommendation to degrade to manual tests is still being discussed on |
| 410 gracefully degrade to manual tests is not currently enforced in code review. | 500 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 411 When considering skipping this recommendation, please keep in mind that a manual | 501 However, please keep in mind that a manual test can be debugged in the browser, |
| 412 test can be debugged in the browser, whereas a test that does not degrade | 502 whereas a test that does not degrade gracefully can only be debugged in the test |
| 413 gracefully can only be debugged in the test runner. Fellow project members and | 503 runner. Fellow project members and future you will thank you for having your |
| 414 future you will thank you for having your test work as a manual test. | 504 test work as a manual test. |
| 415 *** | 505 *** |
| 416 | 506 |
| 417 Manual tests should minimize the chance of user error. This implies keeping the | 507 Manual tests should minimize the chance of user error. This implies keeping the |
| 418 manual steps to a minimum, and having simple and clear instructions that | 508 manual steps to a minimum, and having simple and clear instructions that |
| 419 describe all the configuration changes and user gestures that match the effect | 509 describe all the configuration changes and user gestures that match the effect |
| 420 of the Blink-specific APIs used by the test. | 510 of the Blink-specific APIs used by the test. |
| 421 | 511 |
| 422 Below is an example of a fairly minimal test that uses a Blink-Specific API | 512 Below is an example of a fairly minimal test that uses a Blink-Specific API |
| 423 (`window.eventSender`), and gracefully degrades to a manual test. | 513 (`window.eventSender`), and gracefully degrades to a manual test. |
| 424 | 514 |
| (...skipping 151 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 576 where it is useful to have a test pass when the two images do _not_ match. | 666 where it is useful to have a test pass when the two images do _not_ match. |
| 577 | 667 |
| 578 Reference tests are more difficult to debug than JavaScript tests, and tend to | 668 Reference tests are more difficult to debug than JavaScript tests, and tend to |
| 579 be slower as well. Therefore, they should only be used for functionality that | 669 be slower as well. Therefore, they should only be used for functionality that |
| 580 cannot be covered by JavaScript tests. | 670 cannot be covered by JavaScript tests. |
| 581 | 671 |
| 582 New reference tests should follow the | 672 New reference tests should follow the |
| 583 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The | 673 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The |
| 584 most important points are summarized below. | 674 most important points are summarized below. |
| 585 | 675 |
| 586 * The test page declares the reference page using a `<link rel="match">` or | 676 * 🚧 The test page declares the reference page using a |
| 587 `<link rel="mismatch">`, depending on whether the test passes when the test | 677 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test |
| 588 image matches or does not match the reference image. | 678 passes when the test image matches or does not match the reference image. |
| 589 * The reference page must not use the feature being tested. Otherwise, the test | 679 * The reference page must not use the feature being tested. Otherwise, the test |
| 590 is meaningless. | 680 is meaningless. |
| 591 * The reference page should be as simple as possible, and should not depend on | 681 * The reference page should be as simple as possible, and should not depend on |
| 592 advanced features. Ideally, the reference page should render as intended even | 682 advanced features. Ideally, the reference page should render as intended even |
| 593 on browsers with poor CSS support. | 683 on browsers with poor CSS support. |
| 594 * Reference tests should be self-describing. | 684 * Reference tests should be self-describing. |
| 595 * Reference tests do _not_ include `testharness.js`. | 685 * Reference tests do _not_ include `testharness.js`. |
| 596 | 686 |
| 597 Our testing infrastructure was designed for the | 687 🚧 Our testing infrastructure was designed for the |
| 598 [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink | 688 [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink |
| 599 has inherited. The consequences are summarized below. | 689 has inherited. The consequences are summarized below. |
| 600 | 690 |
| 601 * Each reference page must be in the same directory as its associated test. | 691 * Each reference page must be in the same directory as its associated test. |
| 602 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`), | 692 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`), |
| 603 * The reference page must be named `foo-expected` (e.g., | 693 * The reference page must be named `foo-expected` (e.g., |
| 604 `foo-expected.html`) if the test passes when the two images match. | 694 `foo-expected.html`) if the test passes when the two images match. |
| 605 * The reference page must be named `foo-expected-mismatch` (e.g., | 695 * The reference page must be named `foo-expected-mismatch` (e.g., |
| 606 `foo-expected-mismatch.svg`) if the test passes when the two images do | 696 `foo-expected-mismatch.svg`) if the test passes when the two images do |
| 607 _not_ match. | 697 _not_ match. |
| (...skipping 21 matching lines...) Expand all Loading... |
| 629 <!doctype html> | 719 <!doctype html> |
| 630 <meta charset="utf-8"> | 720 <meta charset="utf-8"> |
| 631 | 721 |
| 632 <ol> | 722 <ol> |
| 633 <li value="3">A</li> | 723 <li value="3">A</li> |
| 634 <li value="2">B</li> | 724 <li value="2">B</li> |
| 635 <li value="1">C</li> | 725 <li value="1">C</li> |
| 636 </ol> | 726 </ol> |
| 637 ``` | 727 ``` |
| 638 | 728 |
| 729 *** promo |
| 730 The method for pointing out a test's reference page is still in flux, and is |
| 731 being discussed on |
| 732 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 733 *** |
| 734 |
| 639 ## Pixel Tests | 735 ## Pixel Tests |
| 640 | 736 |
| 641 `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and | 737 `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and |
| 642 `window.testRunner.dumpDragImage()` create an image result that is associated | 738 `window.testRunner.dumpDragImage()` create an image result that is associated |
| 643 with the test. The image result is compared against an image baseline, which is | 739 with the test. The image result is compared against an image baseline, which is |
| 644 an `-expected.png` file associated with the test, and the test passes if the | 740 an `-expected.png` file associated with the test, and the test passes if the |
| 645 image result is identical to the baseline, according to a pixel-by-pixel | 741 image result is identical to the baseline, according to a pixel-by-pixel |
| 646 comparison. Tests that have image results (and baselines) are called **pixel | 742 comparison. Tests that have image results (and baselines) are called **pixel |
| 647 tests**. | 743 tests**. |
| 648 | 744 |
| 649 Pixel tests should still follow the principles laid out above. Pixel tests pose | 745 Pixel tests should still follow the principles laid out above. Pixel tests pose |
| 650 unique challenges to the desire to have *self-describing* and *cross-platform* | 746 unique challenges to the desire to have *self-describing* and *cross-platform* |
| 651 tests. The | 747 tests. The |
| 652 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli
nes.html) | 748 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli
nes.html) |
| 653 contain useful guidance. The most relevant pieces of advice are below. | 749 contain useful guidance. The most relevant pieces of advice are below. |
| 654 | 750 |
| 655 * Whenever possible, use a green paragraph / page / square to indicate success. | 751 * Whenever possible, use a green paragraph / page / square to indicate success. |
| 656 If that is not possible, make the test self-describing by including a textual | 752 If that is not possible, make the test self-describing by including a textual |
| 657 description of the desired (passing) outcome. | 753 description of the desired (passing) outcome. |
| 658 * Only use the red color or the word `FAIL` to highlight errors. This does not | 754 * Only use the red color or the word `FAIL` to highlight errors. This does not |
| 659 apply when testing the color red. | 755 apply when testing the color red. |
| 660 * Use the [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to | 756 * 🚧 Use the |
| 661 reduce the variance introduced by the platform's text rendering system. This | 757 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the |
| 662 does not apply when testing text, text flow, font selection, font fallback, | 758 variance introduced by the platform's text rendering system. This does not |
| 663 font features, or other typographic information. | 759 apply when testing text, text flow, font selection, font fallback, font |
| 760 features, or other typographic information. |
| 664 | 761 |
| 665 *** promo | 762 *** promo |
| 666 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result | 763 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result |
| 667 will always be 800x600px, because test pages are rendered in an 800x600px | 764 will always be 800x600px, because test pages are rendered in an 800x600px |
| 668 viewport. Pixel tests that do not specifically cover scrolling should fit in an | 765 viewport. Pixel tests that do not specifically cover scrolling should fit in an |
| 669 800x600px viewport without creating scrollbars. | 766 800x600px viewport without creating scrollbars. |
| 670 *** | 767 *** |
| 671 | 768 |
| 769 *** promo |
| 770 The recommendation of using Ahem in pixel tests is being discussed on |
| 771 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| 772 *** |
| 773 |
| 672 The following snippet includes the Ahem font in a layout test. | 774 The following snippet includes the Ahem font in a layout test. |
| 673 | 775 |
| 674 ```html | 776 ```html |
| 675 <style> | 777 <style> |
| 676 body { | 778 body { |
| 677 font: 10px Ahem; | 779 font: 10px Ahem; |
| 678 } | 780 } |
| 679 </style> | 781 </style> |
| 680 <script src="/resources/ahem.js"></script> | 782 <script src="/resources/ahem.js"></script> |
| 681 ``` | 783 ``` |
| (...skipping 96 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 778 * The `http/` directory hosts tests that require an HTTP server (see above). | 880 * The `http/` directory hosts tests that require an HTTP server (see above). |
| 779 * The `resources/` subdirectory in every directory contains binary files, such | 881 * The `resources/` subdirectory in every directory contains binary files, such |
| 780 as media files, and code that is shared by multiple test files. | 882 as media files, and code that is shared by multiple test files. |
| 781 | 883 |
| 782 *** note | 884 *** note |
| 783 Some layout tests consist of a minimal HTML page that references a JavaScript | 885 Some layout tests consist of a minimal HTML page that references a JavaScript |
| 784 file in `resources/`. Please do not use this pattern for new tests, as it goes | 886 file in `resources/`. Please do not use this pattern for new tests, as it goes |
| 785 against the minimality principle. JavaScript and CSS files should only live in | 887 against the minimality principle. JavaScript and CSS files should only live in |
| 786 `resources/` if they are shared by at least two test files. | 888 `resources/` if they are shared by at least two test files. |
| 787 *** | 889 *** |
| OLD | NEW |