Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(37)

Issues Search
    My Issues | Starred     Open | Closed | All

Side by Side Diff: docs/testing/writing_layout_tests.md

Issue 2492733003: New documentation on writing LayoutTests. (Closed)
Patch Set: Addressed feedback from foolip, qyearsley, rbyers. Created 4 years ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « docs/testing/layout_tests.md ('k') | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
(Empty)
1 # Writing Layout Tests
2
3 _Layout tests_ is a bit of a misnomer. This term is
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /),
5 and we use it to refer to every test that is written as a Web page (HTML, SVG,
6 or XHTML) and lives in
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8
9 [TOC]
10
11 ## Overview
12
13 Layout tests should be used to accomplish one of the following goals:
eae 2016/11/22 21:53:11 (optional suggestion) I would change this to say t
pwnall 2016/11/24 03:39:21 It might be the case that I haven't read the right
14
15 1. The entire surface of Blink that is exposed to the Web should be covered by
16 tests that we contribute to the
17 [Web Platform Tests Project](https://github.com/w3c/web-platform-tests)
18 (WPT). This helps us avoid regressions, and helps us identify Web Platform
19 areas where the major browsers don't have interoperable implementations.
20 Furthermore, by contributing to projects such as WPT, we share the burden of
21 writing tests with the other browser vendors, and we help all the browsers
22 get better. This is very much in line with our goal to move the Web forward.
23 2. When a Blink feature cannot be tested using the tools provided by WPT, and
24 cannot be easily covered by
25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs),
26 the feature must be covered by layout tests, to avoid unexpected regressions.
27 These tests will use Blink-specific testing APIs that are only available in
28 [content_shell](./layout_tests_in_content_shell.md).
29
30 *** promo
31 If you know that Blink layout tests are upstreamed to other projects, such as
32 [test262](https://github.com/tc39/test262), please update this document. Most
33 importantly, our guidelines should to make it easy for our tests to be
34 upstreamed. The `blink-dev` mailing list will be happy to help you harmonize our
35 current guidelines with communal test repositories.
36 ***
37
38 ### Test Types
39
40 There are four broad types of layout tests, listed in the order of preference.
41
42 * *JavaScript Tests* are the layout test implementation of
43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
44 assertions written in JavaScript, and pass if the assertions evaluate to
45 true.
46 * *Reference Tests* render a test page and a reference page, and pass if the two
47 renderings are identical, according to a pixel-by-pixel comparison. These
48 tests are less robust, harder to debug, and significantly slower than
49 JavaScript tests, and are only used when JavaScript tests are insufficient,
50 such as when testing paint code.
51 * *Pixel Tests* render a test page and compare the result against a pre-rendered
52 baseline image in the repository. Pixel tests are less robust than all
53 alternatives listed above, because the rendering of a page is influenced by
54 many factors such as the host computer's graphics card and driver, the
55 platform's text rendering system, and various user-configurable operating
56 system settings. For this reason, it is not uncommon for a pixel test to have
57 a different reference image for each platform that Blink is tested on. Pixel
58 tests are least preferred, because the reference images are
59 [quite cumbersome to manage](./layout_test_expectations.md).
60 * *Dump Render Tree (DRT) Tests* work like pixel tests (described above), but
61 also output a textual representation of the render tree, which is the key
62 data structure in Blink's page rendering system. DRT tests pass if the image
63 comparison test (described above) passes, _and_ if their text output matches a
64 text baseline in the repository. These tests are less desirable than all the
65 alternatives, because they depend on a browser implementation detail.
66
eae 2016/11/22 21:53:11 I'd add web platform tests as a distinct fifth cat
pwnall 2016/11/24 03:39:21 I agree with your assessment of the current situat
67 ## General Principles
68
69 The principles below are adapted from
70 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html)
71 and
72 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree).
73
74 * Tests should be as **short** as possible. The page should only include
75 elements that are necessary and relevant to what is being tested.
eae 2016/11/22 21:53:11 I know it comes from the TTWF guidelines but I don
pwnall 2016/11/24 03:39:21 I modified the description to clarify that "short"
76
eae 2016/11/22 21:53:11 (optional) I'd also include the window size guidel
pwnall 2016/11/24 03:39:21 I added some wording for the window size. Thank yo
77 * Tests should be as **fast** as possible. Blink has several thousand layout
78 tests that are run in parallel, and avoiding unnecessary delays is crucial to
79 keeping our Commit Queue in good shape.
80 * Avoid [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API /WindowTimers/setTimeout),
81 as it wastes time on the testing infrastructure. Instead, use specific
82 event handlers, such as
83 [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEve ntHandlers/onload),
84 to decide when to proceed with a test.
85
86 * Tests should be **reliable** and yield consistent results for a given
87 implementation. Flaky tests slow down fellow developers' debugging efforts and
88 the Commit Queue.
89 * `window.setTimeout` is again a primary offender here. Asides from wasting
90 time on a fast system, tests that rely on fixed timeouts can fail when run
91 on systems that are slower than expected.
92 * Follow the guidelines in this
93 [PSA on writing reliable layout tests](https://docs.google.com/document/d/ 1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
94
95 * Tests should be **self-describing**, so that a project member can recognize
96 whether a test passes or fails without having to read the specification of the
97 feature being tested. `testharness.js` makes a test self-describing when used
98 correctly, but tests that degrade to manual tests
99 [must be carefully designed](http://testthewebforward.org/docs/test-style-guid elines.html)
100 to be self-describing.
101
102 * Tests should use the **minimal** set of platform features needed to express
103 the test scenario concisely.
104 * Avoid depending on edge case behavior of features that aren't explicitly
105 covered by the test. For example, except where testing parsing, tests
106 should contain valid markup (no parsing errors).
107 * Prefer JavaScript's
eae 2016/11/22 21:53:11 Why? The google3 js style guide makes the exact *o
pwnall 2016/11/24 03:39:21 FWIW, the google3 style guide changed completely s
108 [===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operator s/Comparison_Operators#Identity_strict_equality_())
109 operator to
110 [==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators /Comparison_Operators#Equality_())
111 so that readers don't have to reason about
112 [type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abst ract-equality-comparison).
113
114 * Tests should be as **cross-platform** as reasonably possible. Avoid
115 assumptions about device type, screen resolution, etc. Unavoidable assumptions
116 should be documented.
117 * When possible, tests should only use Web platform features, as specified
118 in the relevant standards. When the Web platform's APIs are insufficient,
119 tests should prefer to use WPT extended testing APIs, such as
120 `wpt_automation`.
121 * Test pages should use the HTML5 doctype (`<!doctype html>`) unless they
122 are testing the quirks mode.
123 * Tests should be written under the assumption that they will be upstreamed
124 to the WPT project. For example, tests should follow the
125 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
126 * Tests that use Blink-specific testing APIs should feature-test for the
127 presence of the testing APIs and degrade to
128 [manual tests](http://testthewebforward.org/docs/manual-test.html) when
129 the testing APIs are not present. _This is not currently enforced in code
130 review. However, please keep in mind that a manual test can be debugged in
131 the browser, whereas a test that does not degrade gracefully can only be
132 debugged in the test runner._
133
134 * Tests must be **self-contained** and not depend on external network resources.
135 Unless used by multiple test files, CSS and JavaScript should be inlined using
136 `<style>` and `<script>` tags. Content shared by multiple tests should be
137 placed in a `resources/` directory near the tests that share it. See below for
138 using multiple origins in a test.
139
140 * Test **file names** should describe what is being tested. File names should
141 use `snake-case`, but preserve the case of any embedded API names. For
142 example, prefer `document-createElement.html` to
143 `document-create-element.html`.
144
145 * Tests should prefer **modern features** in JavaScript and in the Web Platform.
eae 2016/11/22 21:53:12 These guidelines are in conflict with the *cross-p
foolip 2016/11/23 09:39:23 Some folks, including me, have recently written so
pwnall 2016/11/24 03:39:21 There is a recommendation below for using caniuse.
pwnall 2016/11/24 03:39:21 These guidelines are for new tests. cross-platfor
foolip 2016/11/24 08:52:57 Right, I think that's the best we can do for now.
146 * Tests should use
147 [strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe rence/Strict_mode)
148 for all JavaScript, except when specifically testing sloppy mode behavior.
149 Strict mode flags deprecated features and helps catch some errors, such as
150 forgetting to declare variables.
151 * JavaScript code should prefer
152 [const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statem ents/const)
153 and
154 [let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statemen ts/let)
155 over `var`, prefer
156 [classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Clas ses)
157 over other OOP constructs, and prefer
158 [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Glo bal_Objects/Promise)
159 over other mechanisms for structuring asynchronous code.
160 * The desire to use modern features must be balanced with the desire for
161 cross-platform tests. Avoid using features that haven't shipped by other
162 current major rendering engines (WebKit, Gecko, Edge). When unsure, check
163 [caniuse.com](http://caniuse.com/).
164
165 * Tests must use the UTF-8 **character encoding**, which should be declared by
166 `<meta charset=utf-8>`. This does not apply when specifically testing
167 encodings.
168 * At this time, code reviewers may choose to accept layout tests that do
169 not have a `<meta charset>`, as long as the file contents is pure ASCII.
170 If going that route, please keep in mind that Firefox currently issues a
171 dev tools warning for pages without a declared charset.
172
173 * Tests must aim to have a **coding style** that is consistent with
eae 2016/11/22 21:53:11 Shouldn't this be http://testthewebforward.org/doc
pwnall 2016/11/24 03:39:21 The WPT page describes test construction, not codi
174 [Google's JavaScript Style Guide](https://google.github.io/styleguide/javascri ptguide.xml),
eae 2016/11/22 21:53:11 This has moved. https://google.github.io/styleguid
pwnall 2016/11/24 03:39:21 Done. Thanks so much! FWIW, this wasn't the case w
175 and
176 [Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssgui de.xml),
177 with the following exceptions.
178 * Rules related to Google Closure and JSDoc do not apply.
179 * Modern Web Platform and JavaScript features should be preferred to legacy
eae 2016/11/22 21:53:11 Again, why?
pwnall 2016/11/24 03:39:21 Please see above :)
180 constructs that target old browsers. For example, prefer `const` and `let`
181 to `var`, and prefer `class` over other OOP constructs. This should be
182 balanced with the desire to have cross-platform tests.
183 * Concerns regarding buggy behavior in legacy browsers do not apply. For
184 example, the garbage collection cycle note in the _Closures_ section does
185 not apply.
186 * Per the JavaScript guide, new tests should also follow any per-project
187 style guide, such as the
188 [ServiceWorker Tests Style guide](http://www.chromium.org/blink/servicewor ker/testing).
189
190 ## JavaScript Tests
191
192 Whenever possible, the testing criteria should be expressed in JavaScript. The
193 alternatives, which will be described in future sections, result in slower and
194 less reliable tests.
195
196 All new JavaScript tests should be written using the
197 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This
198 framework is used by the tests in the
199 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
200 which is shared with all the other browser vendors, so `testharness.js` tests
201 are more accessible to browser developers.
202
203 As a shared framework, `testharness.js` enjoys high-quality documentation, such
204 as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and
205 [API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.m d).
206 Layout tests should follow the recommendations of the above documents.
207 Furthermore, layout tests should include relevant
208 [metadata](http://testthewebforward.org/docs/css-metadata.html). The
209 specification URL (in `<link rel="help">`) is almost always relevant, and is
210 incredibly helpful to a developer who needs to understand the test quickly.
211
212 Below is a skeleton for a JavaScript test embedded in an HTML page. Note that,
213 in order to follow the minimality guideline, the test omits the tags `<html>`,
214 `<head>` and `<body>`, as they can be inferred by the HTML parser.
215
216 ```html
217 <!doctype html>
218 <meta charset="utf-8">
219 <title>JavaScript: the true literal</title>
220 <link rel="help" href="https://tc39.github.io/ecma262/#sec-boolean-literals">
221 <meta name="assert" value="The true literal is equal to itself and immutable">
222 <script src="/resources/testharness.js"></script>
223 <script src="/resources/testharnessreport.js"></script>
224 <script>
225 'use strict';
226
227 // Synchronous test example.
228 test(() => {
229 const value = true;
230 assert_true(value, 'true literal');
231 assert_equals(value.toString(), 'true', 'the string representation of true');
232 }, 'The literal true in a synchronous test case');
233
234 // Asynchronous test example.
235 async_test(t => {
236 const originallyTrue = true;
237 setTimeout(t.step_func_done(() => {
238 assert_equals(originallyTrue, true);
239 }), 0);
240 }, 'The literal true in a setTimeout callback');
241
242 // Promise test example.
243 promise_test(() => {
244 return new Promise((resolve, reject) => {
245 resolve(true);
246 }).then(value => {
247 assert_true(value);
248 });
249 }, 'The literal true used to resolve a Promise');
250
251 </script>
252 ```
253
254 Some points that are not immediately obvious from the example:
255
256 * The `<meta name="assert">` describes the purpose of the entire file, and
257 is not redundant to `<title>`. Don't add a `<meta name="assert">` when the
258 information in the `<title>` is sufficient.
259 * When calling an `assert_` function that compares two values, the first
260 argument is the actual value (produced by the functionality being tested), and
261 the second argument is the expected value (known good, golden). The order
262 is important, because the testing harness relies on it to generate expressive
263 error messages that are relied upon when debugging test failures.
264 * The assertion description (the string argument to `assert_` methods) conveys
265 the way the actual value was obtained.
266 * If the expected value doesn't make it clear, the assertion description
267 should explain the desired behavior.
268 * Test cases with a single assertion should omit the assertion's description
269 when it is sufficiently clear.
270 * Each test case describes the circumstance that it tests, without being
271 redundant.
272 * Do not start test case descriptions with redundant terms like "Testing"
273 or "Test for".
274 * Test files with a single test case should omit the test case description.
eae 2016/11/22 21:53:11 This would require tooling changes on our side. T
foolip 2016/11/23 09:23:59 testharness.js uses document.title as the default
275 The file's `<title>` should be sufficient to describe the scenario being
276 tested.
277 * Asynchronous tests have a few subtleties.
278 * The `async_test` wrapper calls its function with a test case argument that
279 is used to signal when the test case is done, and to connect assertion
280 failures to the correct test.
281 * `t.done()` must be called after all the test case's assertions have
282 executed.
283 * Test case assertions (actually, any callback code that can throw
284 exceptions) must be wrapped in `t.step_func()` calls, so that
285 assertion failures and exceptions can be traced back to the correct test
286 case.
287 * `t.step_func_done()` is a shortcut that combines `t.step_func()` with a
288 `t.done()` call.
289
290 *** promo
291 Layout tests that load from `file://` origins must currently use relative paths
292 to point to
293 [/resources/testharness.js](../../third_party/WebKit/LayoutTests/resources/testh arness.js)
294 and
295 [/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources /testharnessreport.js).
296 This is contrary to the WPT guidelines, which call for absolute paths.
297 This limitation does not apply to the tests in `LayoutTests/http`, which rely on
298 an HTTP server, or to the tests in `LayoutTests/imported/wpt`, which are
299 imported from the [WPT repository](https://github.com/w3c/web-platform-tests).
300 ***
301
302 ### WPT Supplemental Testing APIs
303
304 Some tests simply cannot be expressed using the Web Platform APIs. For example,
305 some tests that require a user to perform a gesture, such as a mouse click,
306 cannot be implemented using Web APIs. The WPT project covers some of these cases
307 via supplemental testing APIs.
308
309 *** promo
310 In many cases, the user gesture is not actually necessary. For example, many
311 event handling tests can use
312 [synthetic events](https://developer.mozilla.org/docs/Web/Guide/Events/Creating_ and_triggering_events).
313 ***
314
315 *** note
316 TODO: document wpt_automation. Manual tests might end up moving here.
317 ***
318
319 ### Relying on Blink-Specific Testing APIs
320
321 Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs
322 use Blink-specific testing APIs. These APIs are only available in [content_shell ](./layout_tests_in_content_shell.md), and should only be used as
323 a last resort.
324
325 ### Manual Tests
326
327 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs should =
328 also be usable as
329 [manual tests](http://testthewebforward.org/docs/manual-test.html). This makes
330 it easy to debug the test, and to check whether our behavior matches other
331 browsers.
332
333 *** note
334 The recommendation to have tests that depend on Blink-only testing APIs
335 gracefully degrade to manual tests is not currently enforced in code review.
336 When considering skipping this recommendation, please keep in mind that a manual
337 test can be debugged in the browser, whereas a test that does not degrade
338 gracefully can only be debugged in the test runner. Fellow project members and
339 future you will thank you for having your test work as a manual test.
340 ***
341
342 Manual tests should minimize the chance of user error. This implies keeping the
343 manual steps to a minimum, and having simple and clear instructions that
344 describe all the configuration changes and user gestures that match the effect
345 of the Blink-specific APIs used by the test.
346
347 Below is an example of a fairly minimal test that uses a Blink-Specific API
348 (`window.eventSender`), and gracefully degrades to a manual test.
349
350 ```html
351 <!doctype html>
352 <meta charset="utf-8">
353 <title>DOM: Event.isTrusted for UI events</title>
354 <link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
355 <link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
356 <meta name="assert"
357 content="Event.isTrusted is true for events generated by user interaction">
358 <script src="../../resources/testharness.js"></script>
359 <script src="../../resources/testharnessreport.js"></script>
360
361 <p>Please click on the button below.</p>
362 <button>Click Me!</button>
363
364 <script>
365 'use strict';
366
367 setup({ explicit_timeout: true });
368
369 promise_test(() => {
370 const button = document.querySelector('button');
371 return new Promise((resolve, reject) => {
372 const button = document.querySelector('button');
373 button.addEventListener('click', (event) => {
374 resolve(event);
375 });
376
377 if (window.eventSender) {
378 eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
379 eventSender.mouseDown();
380 eventSender.mouseUp();
381 }
382 }).then((clickEvent) => {
383 assert_true(clickEvent.isTrusted);
384 });
385
386 }, 'Click generated by user interaction');
387
388 </script>
389 ```
390
391 The test exhibits the following desirable features:
392
393 * It has a second specification URL (`<link rel="help">`), because the paragraph
394 that documents the tested feature (referenced by the primary URL) is not very
395 informative on its own.
396 * It links to the
397 [WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_ Standard.22_mean.3F),
398 rather than to a frozen version of the specification.
399 * It contains clear instructions for manually triggering the test conditions.
400 The test starts with a paragraph (`<p>`) that tells the tester exactly what to
401 do, and the `<button>` that needs to be clicked is clearly labeled.
402 * It disables the timeout mechanism built into `testharness.js` by calling
403 `setup({ explicit_timeout: true });`
404 * It checks for the presence of the Blink-specific testing APIs
405 (`window.eventSender`) before invoking them. The test does not automatically
406 fail when the APIs are not present.
407 * It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference /Global_Objects/Promise)
408 to separate the test setup from the assertions. This is particularly helpful
409 for manual tests that depend on a sequence of events to occur, as Promises
410 offer a composable way to express waiting for asynchronous events that avoids
411 [callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).
412
413 Notice that the test is pretty heavy compared to a minimal JavaScript test that
414 does not rely on testing APIs. Only use Blink-specific testing APIs when the
415 desired testing conditions cannot be set up using Web Platform APIs.
416
417 #### Using Blink-Specific Testing APIs
418
419 A downside of Blink-specific APIs is that they are not as well documented as the
420 Web Platform features. Learning to use a Blink-specific feature requires finding
421 other tests that use it, or reading its source code.
422
423 For example, the most popular Blink-specific API is `testRunner`, which is
424 implemented in
425 [components/test_runner/test_runner.h](../../components/test_runner/test_runner. h)
426 and
427 [components/test_runner/test_runner.cpp](../../components/test_runner/test_runne r.cpp).
428 By skimming the `TestRunnerBindings::Install` method, we learn that the
429 testRunner API is presented by the `window.testRunner` and
430 `window.layoutTestsController` objects, which are synonyms. Reading the
431 `TestRunnerBindings::GetObjectTemplateBuilder` method tells us what properties
432 are available on the `window.testRunner` object.
433
434 *** aside
435 `window.testRunner` is the preferred way to access the `testRunner` APIs.
436 `window.layoutTestsController` is still supported because it is used by
437 3rd-party tests.
438 ***
439
440 *** note
441 `testRunner` is the most popular testing API because it is also used indirectly
442 by tests that stick to Web Platform APIs. The `testharnessreport.js` file in
443 `testharness.js` is specifically designated to hold glue code that connects
444 `testharness.js` to the testing environment. Our implementation is in
445 [third_party/WebKit/LayoutTests/resources/testharnessreport.js](../../third_part y/WebKit/LayoutTests/resources/testharnessreport.js),
446 and uses the `testRunner` API.
447 ***
448
449 See the [components/test_runner/](../../components/test_runner/) directory and
450 [WebKit's LayoutTests guide](https://trac.webkit.org/wiki/Writing%20Layout%20Tes ts%20for%20DumpRenderTree)
451 for other useful APIs. For example, `window.eventSender`
452 ([components/test_runner/event_sender.h](../../components/test_runner/event_send er.h)
453 and
454 [components/test_runner/event_sender.cpp](../../components/test_runner/event_sen der.cpp))
455 has methods that simulate events input such as keyboard / mouse input and
456 drag-and-drop.
457
458 Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
459
460 [![UML of testRunner bindings configuring platform implementation](https://docs. google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/sv g?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google .com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
461
462 ### Text Test Baselines
463
464 By default, all the test cases in a file that uses `testharness.js` are expected
465 to pass. However, in some cases, we prefer to add failing test cases to the
466 repository, so that we can be notified when the failure modes change (e.g., we
467 want to know if a test starts crashing rather than returning incorrect output).
468 In these situations, a test file will be accompanied by a baseline, which is an
469 `-expected.txt` file that contains the test's expected output.
470
471 The baselines are generated automatically when appropriate by
472 `run-webkit-tests`, which is described [here](./layout_tests.md), and by the
473 [rebaselining tools](./layout_test_expectations.md).
474
475 Text baselines for `testharness.js` should be avoided, as having a text baseline
476 associated with a `testharness.js` indicates the presence of a bug. For this
477 reason, CLs that add text baselines must include a
478 [crbug.com](https://crbug.com) link for an issue tracking the removal of the
479 text expectations.
480
481 * When creating tests that will be upstreamed to WPT, and Blink's current
482 behavior does not match the specification that is being tested, a text
483 baseline is necessary. Remember to create an issue tracking the expectation's
484 removal, and to link the issue in the CL description.
485 * Layout tests that cannot be upstreamed to WPT should use JavaScript to
486 document Blink's current behavior, rather than using JavaScript to document
487 desired behavior and a text file to document current behavior.
488
489
490 ### The js-test.js Legacy Harness
491
492 *** promo
493 For historical reasons, older tests are written using the `js-test` harness.
494 This harness is **deprecated**, and should not be used for new tests.
495 ***
496
497 If you need to understand old tests, the best `js-test` documentation is its
498 implementation at
499 [third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/L ayoutTests/resources/js-test.js).
500
501 `js-test` tests lean heavily on the Blink-specific `testRunner` testing API.
502 In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page
503 content should be dumped and compared against a text baseline (an
504 `-expected.txt` file). As a consequence, `js-test` tests are always accompanied
505 by text baselines. Asynchronous tests also use `testRunner.waitUntilDone()` and
506 `testRunner.notifyDone()` to tell the testing tools when they are complete.
507
508 ### Tests that use an HTTP Server
509
510 By default, tests are loaded as if via `file:` URLs. Some web platform features
511 require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`)
512 or features restricted to secure protocols.
513
514 HTTP tests are those tests that are under `LayoutTests/http/tests` (or virtual
515 variants). Use a locally running HTTP server (Apache) to run them. Tests are
516 served off of ports 8000 and 8080 for HTTP, and 8443 for HTTPS. If you run the
517 tests using `run-webkit-tests`, the server will be started automatically. To run
518 the server manually to reproduce or debug a failure:
519
520 ```bash
521 cd src/third_party/WebKit/Tools/Scripts
522 run-blink-httpd start
523 ```
524
525 The layout tests will be served from `http://127.0.0.1:8000`. For example, to
526 run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
527 navigate to
528 `http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
529 tests will behave differently if you go to 127.0.0.1 instead of localhost, so
530 use 127.0.0.1.
531
532 To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
533 or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.
534
535 The test server sets up an alias to `LayoutTests/resources` directory. In HTTP
536 tests, you can access the testing framework at e.g.
537 `src="/js-test-resources/js-test.js"`.
538
539 TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
540 position to use it to run layout tests.
541
542 ## Reference Tests (Reftests)
543
544 Reference tests, also known as reftests, perform a pixel-by-pixel comparison
545 between the rendered image of a test page and the rendered image of a reference
546 page. Most reference tests pass if the two images match, but there are cases
547 where it is useful to have a test pass when the two images do _not_ match.
548
549 Reference tests are more difficult to debug than JavaScript tests, and tend to
550 be slower as well. Therefore, they should only be used for functionality that
551 cannot be covered by JavaScript tests.
552
553 New reference tests should follow the
554 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The
555 most important points are summarized below.
556
557 * The test page declares the reference page using a `<link rel="match">` or
558 `<link rel="mismatch">`, depending on whether the test passes when the test
559 image matches or does not match the reference image.
560 * The reference page must not use the feature being tested. Otherwise, the test
561 is meaningless.
562 * The reference page should be as simple as possible, and should not depend on
563 advanced features. Ideally, the reference page should render as intended even
564 on browsers with poor CSS support.
565 * Reference tests should be self-describing.
566 * Reference tests do _not_ include `testharness.js`.
567
568 Our testing infrastructure was designed for the
569 [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink
570 has inherited. The consequences are summarized below.
571
572 * Each reference page must be in the same directory as its associated test.
573 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`),
574 * The reference page must be named `foo-expected` (e.g.,
575 `foo-expected.html`) if the test passes when the two images match.
576 * The reference must be named `foo-expected-mismatch` (e.g.,
577 `foo-expected-mismatch.svg`) if the test passes when the two images do
578 _not_ match.
579 * Multiple references and chained references are not supported.
580
581 The following example demonstrates a reference test for
582 [<ol>'s reversed attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/El ement/ol).
583 The example assumes that the test page is named `ol-reversed.html`.
584
585 ```html
586 <!doctype html>
587 <meta charset="utf-8">
588 <link rel="match" href="ol-reversed-expected.html">
589
590 <ol reversed>
591 <li>A</li>
592 <li>B</li>
593 <li>C</li>
594 </ol>
595 ```
596
597 The reference page, which must be named `ol-reversed-expected.html`, is below.
598
599 ```html
600 <!doctype html>
601 <meta charset="utf-8">
602
603 <ol>
604 <li value="3">A</li>
605 <li value="2">B</li>
606 <li value="1">C</li>
607 </ol>
608 ```
609
610 ## Pixel Tests
611
612 `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and
613 `window.testRunner.dumpDragImage()` create an image result that is associated
614 with the test. The image result is compared against an image baseline, which is
615 an `-expected.png` file associated with the test, and the test passes if the
616 image result is identical to the baseline, according to a pixel-by-pixel
617 comparison. Tests that have image results (and baselines) are called **pixel
618 tests**.
619
620 Pixel tests should still follow the principles laid out above. Pixel tests pose
621 unique challenges to the desire to have *self-describing* and *cross-platform*
622 tests. The
623 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli nes.html)
624 contain useful guidance. The most relevant pieces of advice are below.
625
626 * use a green paragraph / page / square to indicate success
627 * use the red color or the word `FAIL` to highlight errors
628 * use the [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to
eae 2016/11/22 21:53:12 Unless text, text flow, font selection, font fallb
pwnall 2016/11/24 03:39:21 Good point. I added this to the writeup. Thank you
629 minimize the variance introduced by the platform's text rendering system
630
eae 2016/11/22 21:53:11 * The viewport for pixel tests is 800x600, content
pwnall 2016/11/24 03:39:21 Done. This is super-important. Thank you for bring
631 The following snippet includes the Ahem font in a layout test.
632
633 ```html
634 <style>
635 body {
636 font: 10px Ahem;
637 }
638 </style>
639 <script src="/resources/ahem.js"></script>
640 ```
641
642 *** promo
643 Tests outside `LayoutTests/http` and `LayoutTests/imported/wpt` currently need
644 to use a relative path to
645 [/third_party/WebKit/LayoutTests/resources/ahem.js](../../third_party/WebKit/Lay outTests/resources/ahem.js)
646 ***
647
648 ### Tests that need to paint, raster, or draw a frame of intermediate output
649
650 A layout test does not actually draw frames of output until the test exits. If
651 it is required to generate a painted frame, then use
652 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a
653 frame, then call the passed callback. There is also a library at
654 `fast/repaint/resources/text-based-repaint.js` to help with writing paint
655 invalidation and repaint tests.
656
657 ## Dump Render Tree (DRT) Tests
658
659 A Dump Render Tree test renders a web page and produces two results, which are
eae 2016/11/22 21:53:11 A DRT test can produce either text result, pixel r
pwnall 2016/11/24 03:39:21 I've been trying to use the DRT term to only label
660 compared against two baseline files:
661
662 * The image of the rendered page is compared against an `-expected.png` image
663 baseline, using the same method as pixel tests.
664 * A textual representation of Blink's
665 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction)
666 is compared against an `-expected.txt` text baseline.
667
668 A DRT test passes if both its results match their baselines. Like pixel tests,
669 DRT tests are slow, and their baselines depend on platform-specific mechanisms,
eae 2016/11/22 21:53:12 Actually they are much faster than reference tests
pwnall 2016/11/24 03:39:21 Done. I corrected the text. Thank you very much fo
670 so they often require per-platform baselines. Furthermore, DRT tests depend on
671 the render tree data structure, which means that if we replace the render tree
672 data structure, we will need to look at each DRT test and consider whether it is
673 still meaningful.
674
675 For these reasons, DRT tests should **only** be used to cover aspects of the
676 layout code that can only be tested by looking at the render tree. Any
677 combination of the other test types is preferable to a DRT test. DRT tests are
678 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so
679 the repository may have some unfortunate examples of DRT tests.
680
681 The following page is an example of a DRT test.
682
683 ```html
684 <!doctype html>
685 <meta charset="utf-8">
686 <style>
687 p {
688 font: 10px Ahem;
689 color: green;
690 }
691 </style>
692 <script src="/resources/ahem.js"></script>
693
694 <p>This paragraph must be green.</p>
eae 2016/11/22 21:53:11 This is a bad example. This will render as a serie
pwnall 2016/11/24 03:39:21 I tried to put together a better example. This is
695 ```
696
697 The most important aspects of the example are that the test page does not
698 include a testing framework, and that it follows the guidelines for pixel tests.
699 The test page produces the text result below.
700
701 ```
702 layer at (0,0) size 800x600
703 LayoutView at (0,0) size 800x600
704 layer at (0,0) size 800x30
705 LayoutBlockFlow {HTML} at (0,0) size 800x30
706 LayoutBlockFlow {BODY} at (8,10) size 784x10
707 LayoutBlockFlow {P} at (0,0) size 784x10 [color=#008000]
708 LayoutText {#text} at (0,0) size 290x10
709 text run at (0,0) width 290: "This paragraph must be green."
710 ```
711
712 Notice that the test result above depends on the size of the `<p>` text. The
713 test page uses the Ahem font (introduced above), whose main design goal is
714 consistent cross-platform rendering. Had the test used another font, its text
715 baseline would have depended on the fonts installed on the testing computer, and
eae 2016/11/22 21:53:11 Using Ahem does *not* guarantee that there won't b
pwnall 2016/11/24 03:39:21 I thought the guideline discourages it? Relevant t
716 on the platform's font rendering system. Please follow the pixel tests
717 guidelines and write reliable DRT tests!
718
719 WebKit's render tree is described in
720 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/)
721 on WebKit's blog. Some of the concepts there still apply to Blink's render tree.
722
723 ## Directory Structure
724
725 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
726 lacks a strict, formal structure. The following directories have special
727 meaning:
728
729 * The `http/` directory hosts tests that require a HTTP server (see above).
730 * The `resources/` subdirectory in every directory contains binary files, such
731 as media files, and code that is shared by multiple test files.
732
733 *** note
734 Some layout tests consist of a minimal HTML page that references a JavaScript
735 file in `resources/`. Please do not use this pattern for new tests, as it goes
736 against the minimality principle. JavaScript and CSS files should only live in
737 `resources/` if they are shared by at least two test files.
738 ***
OLDNEW
« no previous file with comments | « docs/testing/layout_tests.md ('k') | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698