Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(303)

Side by Side Diff: docs/testing/writing_layout_tests.md

Issue 2548693002: Editorial changes to layout tests writing document. (Closed)
Patch Set: Addressed feedback. Created 4 years ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Writing Layout Tests 1 # Writing Layout Tests
2 2
3 _Layout tests_ is a bit of a misnomer. This term is 3 _Layout tests_ is a bit of a misnomer. This term is
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /),
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, 5 and we use it to refer to every test that is written as a Web page (HTML, SVG,
6 or XHTML) and lives in 6 or XHTML) and lives in
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8 8
9 [TOC] 9 [TOC]
10 10
(...skipping 13 matching lines...) Expand all
24 cannot be easily covered by 24 cannot be easily covered by
25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), 25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs),
26 the feature must be covered by layout tests, to avoid unexpected regressions. 26 the feature must be covered by layout tests, to avoid unexpected regressions.
27 These tests will use Blink-specific testing APIs that are only available in 27 These tests will use Blink-specific testing APIs that are only available in
28 [content_shell](./layout_tests_in_content_shell.md). 28 [content_shell](./layout_tests_in_content_shell.md).
29 29
30 *** promo 30 *** promo
31 If you know that Blink layout tests are upstreamed to other projects, such as 31 If you know that Blink layout tests are upstreamed to other projects, such as
32 [test262](https://github.com/tc39/test262), please update this document. Most 32 [test262](https://github.com/tc39/test262), please update this document. Most
33 importantly, our guidelines should to make it easy for our tests to be 33 importantly, our guidelines should to make it easy for our tests to be
34 upstreamed. The `blink-dev` mailing list will be happy to help you harmonize our 34 upstreamed. The
35 current guidelines with communal test repositories. 35 [blink-dev mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/ blink-dev)
36 will be happy to help you harmonize our current guidelines with communal test
37 repositories.
36 *** 38 ***
37 39
38 ### Test Types 40 ### Test Types
39 41
40 There are four broad types of layout tests, listed in the order of preference. 42 There are four broad types of layout tests, listed in the order of preference.
41 43
42 * *JavaScript Tests* are the layout test implementation of 44 * *JavaScript Tests* are the layout test implementation of
43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain 45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
44 assertions written in JavaScript, and pass if the assertions evaluate to 46 assertions written in JavaScript, and pass if the assertions evaluate to
45 true. 47 true.
(...skipping 20 matching lines...) Expand all
66 match the baselines in the repository. DRT tests are less desirable than all 68 match the baselines in the repository. DRT tests are less desirable than all
67 the alternatives, because they depend on a browser implementation detail. 69 the alternatives, because they depend on a browser implementation detail.
68 70
69 ## General Principles 71 ## General Principles
70 72
71 The principles below are adapted from 73 The principles below are adapted from
72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) 74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html)
73 and 75 and
74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). 76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree).
75 77
76 * Tests should be **concise**, without compromising on the principles below.
77 Every element and piece of code on the page should be necessary and relevant
78 to what is being tested. For example, don't build a fully functional signup
79 form if you only need a text field or a button.
80 * Content needed to satisfy the principles below is considered necessary.
81 For example, it is acceptable and desirable to add elements that make
82 the test self-describing (see below), and to add code that makes the
83 test more reliable (see below).
84 * Content that makes test failures easier to debug is considered necessary
85 (to maintaining a good development speed), and is both acceptable and
86 desirable.
87 * Conciseness is particularly important for reference tests and pixel
88 tests, as the test pages are rendered in an 800x600px viewport. Having
89 content outside the viewport is undesirable because the outside content
90 does not get compared, and because the resulting scrollbars are
91 platform-specific UI widgets, making the test results less reliable.
92
93 * Tests should be as **fast** as possible, without compromising on the
94 principles below. Blink has several thousand layout tests that are run in
95 parallel, and avoiding unnecessary delays is crucial to keeping our Commit
96 Queue in good shape.
97 * Avoid [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API /WindowTimers/setTimeout),
98 as it wastes time on the testing infrastructure. Instead, use specific
99 event handlers, such as
100 [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEve ntHandlers/onload),
101 to decide when to advance to the next step in a test.
102
103 * Tests should be **reliable** and yield consistent results for a given
104 implementation. Flaky tests slow down your fellow developers' debugging
105 efforts and the Commit Queue.
106 * `window.setTimeout` is again a primary offender here. Asides from wasting
107 time on a fast system, tests that rely on fixed timeouts can fail when run
108 on systems that are slower than expected.
109 * Follow the guidelines in this
110 [PSA on writing reliable layout tests](https://docs.google.com/document/d/ 1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
111
112 * Tests should be **self-describing**, so that a project member can recognize
113 whether a test passes or fails without having to read the specification of the
114 feature being tested. `testharness.js` makes a test self-describing when used
115 correctly, but tests that degrade to manual tests
116 [must be carefully designed](http://testthewebforward.org/docs/test-style-guid elines.html)
117 to be self-describing.
118
119 * Tests should require a **minimal** amount of cognitive effort to read and
120 maintain.
121 * Avoid depending on edge case behavior of features that aren't explicitly
122 covered by the test. For example, except where testing parsing, tests
123 should contain valid markup (no parsing errors).
124 * Tests should provide as much relevant information as possible when
125 failing. `testharness.js` tests should prefer
126 [rich assert_ functions](https://github.com/w3c/testharness.js/blob/master /docs/api.md#list-of-assertions)
127 to combining `assert_true()` with a boolean operator. Using appropriate
128 `assert_` functions results in better diagnostic output when the assertion
129 fails.
130 * Prefer JavaScript's
131 [===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operator s/Comparison_Operators#Identity_strict_equality_())
132 operator to
133 [==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators /Comparison_Operators#Equality_())
134 so that readers don't have to reason about
135 [type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abst ract-equality-comparison).
136
137 * Tests should be as **cross-platform** as reasonably possible. Avoid
138 assumptions about device type, screen resolution, etc. Unavoidable assumptions
139 should be documented.
140 * When possible, tests should only use Web platform features, as specified
141 in the relevant standards. When the Web platform's APIs are insufficient,
142 tests should prefer to use WPT extended testing APIs, such as
143 `wpt_automation`.
144 * Test pages should use the HTML5 doctype (`<!doctype html>`) unless they
145 specifically cover
146 [quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards _Mode)
147 behavior.
148 * Tests should be written under the assumption that they will be upstreamed
149 to the WPT project. For example, tests should follow the
150 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
151 * Tests that use Blink-specific testing APIs should feature-test for the
152 presence of the testing APIs and degrade to
153 [manual tests](http://testthewebforward.org/docs/manual-test.html) when
154 the testing APIs are not present. _This is not currently enforced in code
155 review. However, please keep in mind that a manual test can be debugged in
156 the browser, whereas a test that does not degrade gracefully can only be
157 debugged in the test runner._
158
159 * Tests must be **self-contained** and not depend on external network resources.
160 Unless used by multiple test files, CSS and JavaScript should be inlined using
161 `<style>` and `<script>` tags. Content shared by multiple tests should be
162 placed in a `resources/` directory near the tests that share it. See below for
163 using multiple origins in a test.
164
165 * Test **file names** should describe what is being tested. File names should
166 use `snake-case`, but preserve the case of any embedded API names. For
167 example, prefer `document-createElement.html` to
168 `document-create-element.html`.
169
170 * Tests should prefer **modern features** in JavaScript and in the Web Platform.
171 * Tests should use
172 [strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe rence/Strict_mode)
173 for all JavaScript, except when specifically testing sloppy mode behavior.
174 Strict mode flags deprecated features and helps catch some errors, such as
175 forgetting to declare variables.
176 * JavaScript code should prefer
177 [const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statem ents/const)
178 and
179 [let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statemen ts/let)
180 over `var`,
181 [classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Clas ses)
182 over other OOP constructs, and
183 [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Glo bal_Objects/Promise)
184 over other mechanisms for structuring asynchronous code.
185 * The desire to use modern features must be balanced with the desire for
186 cross-platform tests. Avoid using features that haven't been shipped by
187 other developed major rendering engines (WebKit, Gecko, Edge). When
188 unsure, check [caniuse.com](http://caniuse.com/).
189
190 * Tests should use the UTF-8 **character encoding**, which should be declared by
191 `<meta charset=utf-8>`. This does not apply when specifically testing
192 encodings.
193 * At this time, code reviewers may choose to accept layout tests that do
194 not have a `<meta charset>`, as long as the file content is pure ASCII.
195 If going that route, please keep in mind that Firefox currently issues a
196 dev tools warning for pages without a declared charset.
197
198 * Tests should aim to have a **coding style** that is consistent with
199 [Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide. html),
200 and
201 [Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssgui de.xml),
202 with the following exceptions.
203 * Rules related to Google Closure and JSDoc do not apply.
204 * Modern Web Platform and JavaScript features should be preferred to legacy
205 constructs that target old browsers. For example, prefer `const` and `let`
206 to `var`, and prefer `class` over other OOP constructs. This should be
207 balanced with the desire to have cross-platform tests.
208 * Concerns regarding buggy behavior in legacy browsers do not apply. For
209 example, the garbage collection cycle note in the _Closures_ section does
210 not apply.
211 * Per the JavaScript guide, new tests should also follow any per-project
212 style guide, such as the
213 [ServiceWorker Tests Style guide](http://www.chromium.org/blink/servicewor ker/testing).
214
215 *** note 78 *** note
216 This document intentionally uses _should_ a lot more than _must_, as defined in 79 This document intentionally uses _should_ a lot more than _must_, as defined in
217 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a 80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a
218 careful act of balancing many concerns, and this humble document cannot possibly 81 careful act of balancing many concerns, and this humble document cannot possibly
219 capture the context that rests in the head of an experienced Blink engineer. 82 capture the context that rests in the head of an experienced Blink engineer.
220 *** 83 ***
221 84
85 ### Concise
86
87 Tests should be **concise**, without compromising on the principles below. Every
88 element and piece of code on the page should be necessary and relevant to what
89 is being tested. For example, don't build a fully functional signup form if you
90 only need a text field or a button.
91
92 Content needed to satisfy the principles below is considered necessary. For
93 example, it is acceptable and desirable to add elements that make the test
94 self-describing (see below), and to add code that makes the test more reliable
95 (see below).
96
97 Content that makes test failures easier to debug is considered necessary (to
98 maintaining a good development speed), and is both acceptable and desirable.
99
100 *** promo
101 Conciseness is particularly important for reference tests and pixel tests, as
102 the test pages are rendered in an 800x600px viewport. Having content outside the
103 viewport is undesirable because the outside content does not get compared, and
104 because the resulting scrollbars are platform-specific UI widgets, making the
105 test results less reliable.
106 ***
107
108 ### Fast
109
110 Tests should be as **fast** as possible, without compromising on the principles
111 below. Blink has several thousand layout tests that are run in parallel, and
112 avoiding unnecessary delays is crucial to keeping our Commit Queue in good
113 shape.
114
115 Avoid
116 [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimer s/setTimeout),
117 as it wastes time on the testing infrastructure. Instead, use specific event
118 handlers, such as
119 [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHand lers/onload),
120 to decide when to advance to the next step in a test.
121
122 ### Reliable
123
124 Tests should be **reliable** and yield consistent results for a given
125 implementation. Flaky tests slow down your fellow developers' debugging efforts
126 and the Commit Queue.
127
128 `window.setTimeout` is again a primary offender here. Asides from wasting time
129 on a fast system, tests that rely on fixed timeouts can fail when on systems
130 that are slower than expected.
131
132 Follow the guidelines in this
133 [PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4Sn TLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
134
135 ### Self-Describing
136
137 Tests should be **self-describing**, so that a project member can recognize
138 whether a test passes or fails without having to read the specification of the
139 feature being tested. `testharness.js` makes a test self-describing when used
140 correctly, but tests that degrade to manual tests
141 [must be carefully designed](http://testthewebforward.org/docs/test-style-guidel ines.html)
142 to be self-describing.
143
144 ### Minimal
145
146 Tests should require a **minimal** amount of cognitive effort to read and
147 maintain.
148
149 Avoid depending on edge case behavior of features that aren't explicitly covered
150 by the test. For example, except where testing parsing, tests should contain
151 valid markup (no parsing errors).
152
153 Tests should provide as much relevant information as possible when failing.
154 `testharness.js` tests should prefer
155 [rich assert_ functions](https://github.com/w3c/testharness.js/blob/master/docs/ api.md#list-of-assertions)
156 to combining `assert_true()` with a boolean operator. Using appropriate
157 `assert_` functions results in better diagnostic output when the assertion
158 fails.
159
160 &#x1F6A7; Prefer JavaScript's
161 [===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comp arison_Operators#Identity_strict_equality_())
162 operator to
163 [==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Compa rison_Operators#Equality_())
164 so that readers don't have to reason about
165 [type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abstract-e quality-comparison).
166
167 *** promo
168 The === vs == recommendation is still being discussed on
169 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
170 However, please keep in mind that a fellow developer who has to debug a failing
171 test may have to consider the
172 [special cases for ==](http://dorey.github.io/JavaScript-Equality-Table/) when
173 the types of the two values being compared aren't immediately obvious.
174 ***
175
176 ### Cross-Platform
177
178 Tests should be as **cross-platform** as reasonably possible. Avoid assumptions
179 about device type, screen resolution, etc. Unavoidable assumptions should be
180 documented.
181
182 When possible, tests should only use Web platform features, as specified
183 in the relevant standards. When the Web platform's APIs are insufficient,
184 tests should prefer to use WPT extended testing APIs, such as
185 `wpt_automation`, over Blink-specific testing APIs.
186
187 &#x1F6A7; Tests that use testing APIs should feature-test for the presence of
188 those APIs, and gracefully degrade to manual tests (see below) when the testing
189 APIs are not available.
190
191 Test pages should use the HTML5 doctype (`<!doctype html>`) unless they
192 specifically cover
193 [quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards_Mode)
194 behavior.
195
196 Tests should be written under the assumption that they will be upstreamed
197 to the WPT project. For example, tests should follow the
198 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
199
200 Tests should avoid using features that haven't been shipped by the
201 actively-developed major rendering engines (Blink, WebKit, Gecko, Edge). When
202 unsure, check [caniuse.com](http://caniuse.com/). By necessity, this
203 recommendation does not apply to the feature targeted by the test.
204
205 *** note
206 It may be tempting have a test for a bleeding-edge feature X depend on feature
207 Y, which has only shipped in beta / development versions of various browsers.
208 The reasoning would be that all browsers that implement X will have implemented
209 Y. Please keep in mind that Chrome has un-shipped features that made it to the
210 Beta channel in the past.
211 ***
212
213 Tests that use Blink-specific testing APIs should feature-test for the
214 presence of the testing APIs and degrade to
215 [manual tests](http://testthewebforward.org/docs/manual-test.html) when
216 the testing APIs are not present.
217
218 *** promo
219 The recommendation to degrade to manual tests is still being discussed on
220 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
221 However, please keep in mind that a manual test can be debugged in the browser,
222 whereas a test that does not degrade gracefully can only be debugged in the test
223 runner. Fellow project members and future you will thank you for having your
224 test work as a manual test.
225 ***
226
227 ### Self-Contained
228
229 Tests must be **self-contained** and not depend on external network resources.
230
231 Unless used by multiple test files, CSS and JavaScript should be inlined using
232 `<style>` and `<script>` tags. Content shared by multiple tests should be
233 placed in a `resources/` directory near the tests that share it. See below for
234 using multiple origins in a test.
235
236 ### File Names
237
238 Test **file names** should describe what is being tested.
239
240 File names should use `snake-case`, but preserve the case of any embedded API
241 names. For example, prefer `document-createElement.html` to
242 `document-create-element.html`.
243
244 ### Modern Features
245
246 Tests should prefer **modern features** in JavaScript and in the Web Platform,
247 provided that they meet the recommendations above for cross-platform tests.
248
249 Tests should use
250 [strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/ Strict_mode)
251 for all JavaScript, except when specifically testing sloppy mode behavior.
252 Strict mode flags deprecated features and helps catch some errors, such as
253 forgetting to declare variables.
254
255 &#x1F6A7; JavaScript code should prefer
256 [const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/c onst)
257 and
258 [let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/let )
259 over `var`,
260 [classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Classes)
261 over other OOP constructs, and
262 [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Ob jects/Promise)
263 over other mechanisms for structuring asynchronous code.
264
265 *** promo
266 The recommendation to prefer `const` and `let` over `var` is currently being
267 discussed on
268 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
269 ***
270
271 ### Character Encoding
272
273 Tests should use the UTF-8 **character encoding**, which should be declared by
274 `<meta charset=utf-8>`. This does not apply when specifically testing encodings.
275
276 The `<meta>` tag must be the first child of the document's `<head>` element. In
277 documents that do not have an explicit `<head>`, the `<meta>` tag must follow
278 the doctype.
279
280 When HTML pages do not explicitly declare a character encoding, browsers
281 determine the encoding using an
282 [encoding sniffing algorithm](https://html.spec.whatwg.org/multipage/syntax.html #determining-the-character-encoding)
283 that will surprise most modern Web developers. Highlights include a default
284 encoding that depends on the user's locale, and non-standardized
285 browser-specific heuristics.
286
287 *** promo
288 The WPT guidelines state that test files that only contain ASCII characters may
289 omit the `<meta>` tag. This exception is currently discussed on
290 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
291 If taking that route, please keep in mind that Firefox currently issues a
292 [development tools](https://developer.mozilla.org/en-US/docs/Tools) warning for
293 pages without a declared encoding.
294 ***
295
296 ### Coding Style
297
298 Tests should aim to have a **coding style** that is consistent with
299 [Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.ht ml),
300 and
301 [Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide .xml),
302 with the following exceptions.
303
304 * Rules related to Google Closure and JSDoc do not apply.
305 * Modern Web Platform and JavaScript features should be preferred to legacy
306 constructs that target old browsers.
307 * Per the JavaScript guide, new tests should also follow any per-project
308 style guide, such as the
309 [ServiceWorker Tests Style guide](http://www.chromium.org/blink/serviceworker/ testing).
310
222 ## JavaScript Tests 311 ## JavaScript Tests
223 312
224 Whenever possible, the testing criteria should be expressed in JavaScript. The 313 Whenever possible, the testing criteria should be expressed in JavaScript. The
225 alternatives, which will be described in future sections, result in slower and 314 alternatives, which will be described in future sections, result in slower and
226 less reliable tests. 315 less reliable tests.
227 316
228 All new JavaScript tests should be written using the 317 All new JavaScript tests should be written using the
229 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This 318 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This
230 framework is used by the tests in the 319 framework is used by the tests in the
231 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, 320 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
(...skipping 158 matching lines...) Expand 10 before | Expand all | Expand 10 after
390 for other useful APIs. For example, `window.eventSender` 479 for other useful APIs. For example, `window.eventSender`
391 ([components/test_runner/event_sender.h](../../components/test_runner/event_send er.h) 480 ([components/test_runner/event_sender.h](../../components/test_runner/event_send er.h)
392 and 481 and
393 [components/test_runner/event_sender.cpp](../../components/test_runner/event_sen der.cpp)) 482 [components/test_runner/event_sender.cpp](../../components/test_runner/event_sen der.cpp))
394 has methods that simulate events input such as keyboard / mouse input and 483 has methods that simulate events input such as keyboard / mouse input and
395 drag-and-drop. 484 drag-and-drop.
396 485
397 Here is a UML diagram of how the `testRunner` bindings fit into Chromium. 486 Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
398 487
399 [![UML of testRunner bindings configuring platform implementation](https://docs. google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/sv g?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google .com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) 488 [![UML of testRunner bindings configuring platform implementation](https://docs. google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/sv g?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google .com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
489
400 ### Manual Tests 490 ### Manual Tests
401 491
402 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs should 492 &#x1F6A7; Whenever possible, tests that rely on (WPT's or Blink's) testing APIs
403 also be usable as 493 should also be usable as
404 [manual tests](http://testthewebforward.org/docs/manual-test.html). This makes 494 [manual tests](http://testthewebforward.org/docs/manual-test.html). This makes
405 it easy to debug the test, and to check whether our behavior matches other 495 it easy to debug the test, and to check whether our behavior matches other
406 browsers. 496 browsers.
407 497
408 *** note 498 *** promo
409 The recommendation to have tests that depend on Blink-only testing APIs 499 The recommendation to degrade to manual tests is still being discussed on
410 gracefully degrade to manual tests is not currently enforced in code review. 500 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
411 When considering skipping this recommendation, please keep in mind that a manual 501 However, please keep in mind that a manual test can be debugged in the browser,
412 test can be debugged in the browser, whereas a test that does not degrade 502 whereas a test that does not degrade gracefully can only be debugged in the test
413 gracefully can only be debugged in the test runner. Fellow project members and 503 runner. Fellow project members and future you will thank you for having your
414 future you will thank you for having your test work as a manual test. 504 test work as a manual test.
415 *** 505 ***
416 506
417 Manual tests should minimize the chance of user error. This implies keeping the 507 Manual tests should minimize the chance of user error. This implies keeping the
418 manual steps to a minimum, and having simple and clear instructions that 508 manual steps to a minimum, and having simple and clear instructions that
419 describe all the configuration changes and user gestures that match the effect 509 describe all the configuration changes and user gestures that match the effect
420 of the Blink-specific APIs used by the test. 510 of the Blink-specific APIs used by the test.
421 511
422 Below is an example of a fairly minimal test that uses a Blink-Specific API 512 Below is an example of a fairly minimal test that uses a Blink-Specific API
423 (`window.eventSender`), and gracefully degrades to a manual test. 513 (`window.eventSender`), and gracefully degrades to a manual test.
424 514
(...skipping 151 matching lines...) Expand 10 before | Expand all | Expand 10 after
576 where it is useful to have a test pass when the two images do _not_ match. 666 where it is useful to have a test pass when the two images do _not_ match.
577 667
578 Reference tests are more difficult to debug than JavaScript tests, and tend to 668 Reference tests are more difficult to debug than JavaScript tests, and tend to
579 be slower as well. Therefore, they should only be used for functionality that 669 be slower as well. Therefore, they should only be used for functionality that
580 cannot be covered by JavaScript tests. 670 cannot be covered by JavaScript tests.
581 671
582 New reference tests should follow the 672 New reference tests should follow the
583 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The 673 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The
584 most important points are summarized below. 674 most important points are summarized below.
585 675
586 * The test page declares the reference page using a `<link rel="match">` or 676 * &#x1F6A7; The test page declares the reference page using a
587 `<link rel="mismatch">`, depending on whether the test passes when the test 677 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test
588 image matches or does not match the reference image. 678 passes when the test image matches or does not match the reference image.
589 * The reference page must not use the feature being tested. Otherwise, the test 679 * The reference page must not use the feature being tested. Otherwise, the test
590 is meaningless. 680 is meaningless.
591 * The reference page should be as simple as possible, and should not depend on 681 * The reference page should be as simple as possible, and should not depend on
592 advanced features. Ideally, the reference page should render as intended even 682 advanced features. Ideally, the reference page should render as intended even
593 on browsers with poor CSS support. 683 on browsers with poor CSS support.
594 * Reference tests should be self-describing. 684 * Reference tests should be self-describing.
595 * Reference tests do _not_ include `testharness.js`. 685 * Reference tests do _not_ include `testharness.js`.
596 686
597 Our testing infrastructure was designed for the 687 &#x1F6A7; Our testing infrastructure was designed for the
598 [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink 688 [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink
599 has inherited. The consequences are summarized below. 689 has inherited. The consequences are summarized below.
600 690
601 * Each reference page must be in the same directory as its associated test. 691 * Each reference page must be in the same directory as its associated test.
602 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`), 692 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`),
603 * The reference page must be named `foo-expected` (e.g., 693 * The reference page must be named `foo-expected` (e.g.,
604 `foo-expected.html`) if the test passes when the two images match. 694 `foo-expected.html`) if the test passes when the two images match.
605 * The reference page must be named `foo-expected-mismatch` (e.g., 695 * The reference page must be named `foo-expected-mismatch` (e.g.,
606 `foo-expected-mismatch.svg`) if the test passes when the two images do 696 `foo-expected-mismatch.svg`) if the test passes when the two images do
607 _not_ match. 697 _not_ match.
(...skipping 21 matching lines...) Expand all
629 <!doctype html> 719 <!doctype html>
630 <meta charset="utf-8"> 720 <meta charset="utf-8">
631 721
632 <ol> 722 <ol>
633 <li value="3">A</li> 723 <li value="3">A</li>
634 <li value="2">B</li> 724 <li value="2">B</li>
635 <li value="1">C</li> 725 <li value="1">C</li>
636 </ol> 726 </ol>
637 ``` 727 ```
638 728
729 *** promo
730 The method for pointing out a test's reference page is still in flux, and is
731 being discussed on
732 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
733 ***
734
639 ## Pixel Tests 735 ## Pixel Tests
640 736
641 `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and 737 `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and
642 `window.testRunner.dumpDragImage()` create an image result that is associated 738 `window.testRunner.dumpDragImage()` create an image result that is associated
643 with the test. The image result is compared against an image baseline, which is 739 with the test. The image result is compared against an image baseline, which is
644 an `-expected.png` file associated with the test, and the test passes if the 740 an `-expected.png` file associated with the test, and the test passes if the
645 image result is identical to the baseline, according to a pixel-by-pixel 741 image result is identical to the baseline, according to a pixel-by-pixel
646 comparison. Tests that have image results (and baselines) are called **pixel 742 comparison. Tests that have image results (and baselines) are called **pixel
647 tests**. 743 tests**.
648 744
649 Pixel tests should still follow the principles laid out above. Pixel tests pose 745 Pixel tests should still follow the principles laid out above. Pixel tests pose
650 unique challenges to the desire to have *self-describing* and *cross-platform* 746 unique challenges to the desire to have *self-describing* and *cross-platform*
651 tests. The 747 tests. The
652 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli nes.html) 748 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli nes.html)
653 contain useful guidance. The most relevant pieces of advice are below. 749 contain useful guidance. The most relevant pieces of advice are below.
654 750
655 * Whenever possible, use a green paragraph / page / square to indicate success. 751 * Whenever possible, use a green paragraph / page / square to indicate success.
656 If that is not possible, make the test self-describing by including a textual 752 If that is not possible, make the test self-describing by including a textual
657 description of the desired (passing) outcome. 753 description of the desired (passing) outcome.
658 * Only use the red color or the word `FAIL` to highlight errors. This does not 754 * Only use the red color or the word `FAIL` to highlight errors. This does not
659 apply when testing the color red. 755 apply when testing the color red.
660 * Use the [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to 756 * &#x1F6A7; Use the
661 reduce the variance introduced by the platform's text rendering system. This 757 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the
662 does not apply when testing text, text flow, font selection, font fallback, 758 variance introduced by the platform's text rendering system. This does not
663 font features, or other typographic information. 759 apply when testing text, text flow, font selection, font fallback, font
760 features, or other typographic information.
664 761
665 *** promo 762 *** promo
666 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result 763 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result
667 will always be 800x600px, because test pages are rendered in an 800x600px 764 will always be 800x600px, because test pages are rendered in an 800x600px
668 viewport. Pixel tests that do not specifically cover scrolling should fit in an 765 viewport. Pixel tests that do not specifically cover scrolling should fit in an
669 800x600px viewport without creating scrollbars. 766 800x600px viewport without creating scrollbars.
670 *** 767 ***
671 768
769 *** promo
770 The recommendation of using Ahem in pixel tests is being discussed on
771 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
772 ***
773
672 The following snippet includes the Ahem font in a layout test. 774 The following snippet includes the Ahem font in a layout test.
673 775
674 ```html 776 ```html
675 <style> 777 <style>
676 body { 778 body {
677 font: 10px Ahem; 779 font: 10px Ahem;
678 } 780 }
679 </style> 781 </style>
680 <script src="/resources/ahem.js"></script> 782 <script src="/resources/ahem.js"></script>
681 ``` 783 ```
(...skipping 96 matching lines...) Expand 10 before | Expand all | Expand 10 after
778 * The `http/` directory hosts tests that require an HTTP server (see above). 880 * The `http/` directory hosts tests that require an HTTP server (see above).
779 * The `resources/` subdirectory in every directory contains binary files, such 881 * The `resources/` subdirectory in every directory contains binary files, such
780 as media files, and code that is shared by multiple test files. 882 as media files, and code that is shared by multiple test files.
781 883
782 *** note 884 *** note
783 Some layout tests consist of a minimal HTML page that references a JavaScript 885 Some layout tests consist of a minimal HTML page that references a JavaScript
784 file in `resources/`. Please do not use this pattern for new tests, as it goes 886 file in `resources/`. Please do not use this pattern for new tests, as it goes
785 against the minimality principle. JavaScript and CSS files should only live in 887 against the minimality principle. JavaScript and CSS files should only live in
786 `resources/` if they are shared by at least two test files. 888 `resources/` if they are shared by at least two test files.
787 *** 889 ***
OLDNEW
« no previous file with comments | « no previous file | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698