OLD | NEW |
---|---|
1 # Writing Layout Tests | 1 # Writing Layout Tests |
2 | 2 |
3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), |
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
6 or XHTML) and lives in | 6 or XHTML) and lives in |
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
8 | 8 |
9 [TOC] | 9 [TOC] |
10 | 10 |
11 ## Overview | 11 ## Overview |
12 | 12 |
13 Layout tests should be used to accomplish one of the following goals: | 13 Layout tests should be used to accomplish one of the following goals: |
14 | 14 |
15 1. The entire surface of Blink that is exposed to the Web should be covered by | 15 1. The entire surface of Blink that is exposed to the Web should be covered by |
16 tests that we contribute to the | 16 tests that we contribute to [web-platform-tests](./web_platform_tests.md) |
17 [Web Platform Tests Project](./web_platform_tests.md) (WPT). This helps us | 17 (WPT). This helps us avoid regressions, and helps us identify Web Platform |
pwnall
2017/02/16 19:10:40
do we still use WPT, or should we use wpt for web-
foolip
2017/02/16 19:18:58
I have consulted my inbox and see people using bot
| |
18 avoid regressions, and helps us identify Web Platform areas where the major | 18 areas where the major browsers don't have interoperable implementations. |
19 browsers don't have interoperable implementations. Furthermore, by | 19 Furthermore, by contributing to projects such as WPT, we share the burden of |
20 contributing to projects such as WPT, we share the burden of writing tests | 20 writing tests with the other browser vendors, and we help all the browsers |
21 with the other browser vendors, and we help all the browsers get better. This | 21 get better. This is very much in line with our goal to move the Web forward. |
22 is very much in line with our goal to move the Web forward. | |
23 2. When a Blink feature cannot be tested using the tools provided by WPT, and | 22 2. When a Blink feature cannot be tested using the tools provided by WPT, and |
24 cannot be easily covered by | 23 cannot be easily covered by |
25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), | 24 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), |
26 the feature must be covered by layout tests, to avoid unexpected regressions. | 25 the feature must be covered by layout tests, to avoid unexpected regressions. |
27 These tests will use Blink-specific testing APIs that are only available in | 26 These tests will use Blink-specific testing APIs that are only available in |
28 [content_shell](./layout_tests_in_content_shell.md). | 27 [content_shell](./layout_tests_in_content_shell.md). |
29 | 28 |
30 *** promo | 29 *** promo |
31 If you know that Blink layout tests are upstreamed to other projects, such as | 30 If you know that Blink layout tests are upstreamed to other projects, such as |
32 [test262](https://github.com/tc39/test262), please update this document. Most | 31 [test262](https://github.com/tc39/test262), please update this document. Most |
(...skipping 34 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
67 test passes if the output matches a baseline text file in the repository. | 66 test passes if the output matches a baseline text file in the repository. |
68 Layout tree tests are used as a last resort to test the internal quirks of | 67 Layout tree tests are used as a last resort to test the internal quirks of |
69 the implementation, and they should be avoided in favor of one of the earlier | 68 the implementation, and they should be avoided in favor of one of the earlier |
70 options. | 69 options. |
71 | 70 |
72 ## General Principles | 71 ## General Principles |
73 | 72 |
74 | 73 |
75 Tests should be written under the assumption that they will be upstreamed | 74 Tests should be written under the assumption that they will be upstreamed |
76 to the WPT project. To this end, tests should follow the | 75 to the WPT project. To this end, tests should follow the |
77 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). | 76 [WPT guidelines](http://web-platform-tests.org/writing-tests/). |
78 | 77 |
79 | 78 |
80 There is no style guide that applies to all layout tests. However, some projects | 79 There is no style guide that applies to all layout tests. However, some projects |
81 have adopted style guides, such as the | 80 have adopted style guides, such as the |
82 [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/t esting). | 81 [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/t esting). |
83 | 82 |
84 Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most | 83 Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most |
85 important WPT guidelines and highlights some JavaScript concepts that are worth | 84 important WPT guidelines and highlights some JavaScript concepts that are worth |
86 paying attention to when trying to infer style rules from existing tests. If | 85 paying attention to when trying to infer style rules from existing tests. If |
87 you're unopinionated and looking for a style guide to follow, the document also | 86 you're unopinionated and looking for a style guide to follow, the document also |
88 suggests some defaults. | 87 suggests some defaults. |
89 | 88 |
90 ## JavaScript Tests | 89 ## JavaScript Tests |
91 | 90 |
92 Whenever possible, the testing criteria should be expressed in JavaScript. The | 91 Whenever possible, the testing criteria should be expressed in JavaScript. The |
93 alternatives, which will be described in future sections, result in slower and | 92 alternatives, which will be described in future sections, result in slower and |
94 less reliable tests. | 93 less reliable tests. |
95 | 94 |
96 All new JavaScript tests should be written using the | 95 All new JavaScript tests should be written using the |
97 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This | 96 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This |
98 framework is used by the tests in the | 97 framework is used by the tests in the |
99 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, | 98 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, |
100 which is shared with all the other browser vendors, so `testharness.js` tests | 99 which is shared with all the other browser vendors, so `testharness.js` tests |
101 are more accessible to browser developers. | 100 are more accessible to browser developers. |
102 | 101 |
103 As a shared framework, `testharness.js` enjoys high-quality documentation, such | 102 See the [API documentation](https://github.com/w3c/testharness.js/blob/master/do cs/api.md) |
104 as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and | 103 for a thorough introduction to `testharness.js`. |
105 [API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.m d). | 104 |
106 Layout tests should follow the recommendations of the above documents. | 105 Layout tests should follow the recommendations of the above documentation. |
107 Furthermore, layout tests should include relevant | 106 Furthermore, layout tests should include relevant |
108 [metadata](http://testthewebforward.org/docs/css-metadata.html). The | 107 [metadata](http://web-platform-tests.org/writing-tests/css-metadata.html). The |
109 specification URL (in `<link rel="help">`) is almost always relevant, and is | 108 specification URL (in `<link rel="help">`) is almost always relevant, and is |
110 incredibly helpful to a developer who needs to understand the test quickly. | 109 incredibly helpful to a developer who needs to understand the test quickly. |
111 | 110 |
112 Below is a skeleton for a JavaScript test embedded in an HTML page. Note that, | 111 Below is a skeleton for a JavaScript test embedded in an HTML page. Note that, |
113 in order to follow the minimality guideline, the test omits the tags `<html>`, | 112 in order to follow the minimality guideline, the test omits the tags `<html>`, |
114 `<head>`, and `<body>`, as they can be inferred by the HTML parser. | 113 `<head>`, and `<body>`, as they can be inferred by the HTML parser. |
115 | 114 |
116 ```html | 115 ```html |
117 <!doctype html> | 116 <!doctype html> |
118 <title>JavaScript: the true literal is immutable and equal to itself</title> | 117 <title>JavaScript: the true literal is immutable and equal to itself</title> |
(...skipping 227 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
346 Reference tests, also known as reftests, perform a pixel-by-pixel comparison | 345 Reference tests, also known as reftests, perform a pixel-by-pixel comparison |
347 between the rendered image of a test page and the rendered image of a reference | 346 between the rendered image of a test page and the rendered image of a reference |
348 page. Most reference tests pass if the two images match, but there are cases | 347 page. Most reference tests pass if the two images match, but there are cases |
349 where it is useful to have a test pass when the two images do _not_ match. | 348 where it is useful to have a test pass when the two images do _not_ match. |
350 | 349 |
351 Reference tests are more difficult to debug than JavaScript tests, and tend to | 350 Reference tests are more difficult to debug than JavaScript tests, and tend to |
352 be slower as well. Therefore, they should only be used for functionality that | 351 be slower as well. Therefore, they should only be used for functionality that |
353 cannot be covered by JavaScript tests. | 352 cannot be covered by JavaScript tests. |
354 | 353 |
355 New reference tests should follow the | 354 New reference tests should follow the |
356 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The | 355 [WPT reftests guidelines](http://web-platform-tests.org/writing-tests/reftests.h tml). |
357 most important points are summarized below. | 356 The most important points are summarized below. |
358 | 357 |
359 * 🚧 The test page declares the reference page using a | 358 * 🚧 The test page declares the reference page using a |
360 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test | 359 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test |
361 passes when the test image matches or does not match the reference image. | 360 passes when the test image matches or does not match the reference image. |
362 * The reference page must not use the feature being tested. Otherwise, the test | 361 * The reference page must not use the feature being tested. Otherwise, the test |
363 is meaningless. | 362 is meaningless. |
364 * The reference page should be as simple as possible, and should not depend on | 363 * The reference page should be as simple as possible, and should not depend on |
365 advanced features. Ideally, the reference page should render as intended even | 364 advanced features. Ideally, the reference page should render as intended even |
366 on browsers with poor CSS support. | 365 on browsers with poor CSS support. |
367 * Reference tests should be self-describing. | 366 * Reference tests should be self-describing. |
(...skipping 51 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
419 `window.testRunner.dumpDragImage()` create an image result that is associated | 418 `window.testRunner.dumpDragImage()` create an image result that is associated |
420 with the test. The image result is compared against an image baseline, which is | 419 with the test. The image result is compared against an image baseline, which is |
421 an `-expected.png` file associated with the test, and the test passes if the | 420 an `-expected.png` file associated with the test, and the test passes if the |
422 image result is identical to the baseline, according to a pixel-by-pixel | 421 image result is identical to the baseline, according to a pixel-by-pixel |
423 comparison. Tests that have image results (and baselines) are called **pixel | 422 comparison. Tests that have image results (and baselines) are called **pixel |
424 tests**. | 423 tests**. |
425 | 424 |
426 Pixel tests should still follow the principles laid out above. Pixel tests pose | 425 Pixel tests should still follow the principles laid out above. Pixel tests pose |
427 unique challenges to the desire to have *self-describing* and *cross-platform* | 426 unique challenges to the desire to have *self-describing* and *cross-platform* |
428 tests. The | 427 tests. The |
429 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli nes.html) | 428 [WPT rendering test guidelines](http://web-platform-tests.org/writing-tests/rend ering.html) |
430 contain useful guidance. The most relevant pieces of advice are below. | 429 contain useful guidance. The most relevant pieces of advice are below. |
431 | 430 |
432 * Whenever possible, use a green paragraph / page / square to indicate success. | 431 * Whenever possible, use a green paragraph / page / square to indicate success. |
433 If that is not possible, make the test self-describing by including a textual | 432 If that is not possible, make the test self-describing by including a textual |
434 description of the desired (passing) outcome. | 433 description of the desired (passing) outcome. |
435 * Only use the red color or the word `FAIL` to highlight errors. This does not | 434 * Only use the red color or the word `FAIL` to highlight errors. This does not |
436 apply when testing the color red. | 435 apply when testing the color red. |
437 * 🚧 Use the | 436 * 🚧 Use the |
438 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the | 437 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the |
439 variance introduced by the platform's text rendering system. This does not | 438 variance introduced by the platform's text rendering system. This does not |
(...skipping 135 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
575 * The `http/` directory hosts tests that require an HTTP server (see above). | 574 * The `http/` directory hosts tests that require an HTTP server (see above). |
576 * The `resources/` subdirectory in every directory contains binary files, such | 575 * The `resources/` subdirectory in every directory contains binary files, such |
577 as media files, and code that is shared by multiple test files. | 576 as media files, and code that is shared by multiple test files. |
578 | 577 |
579 *** note | 578 *** note |
580 Some layout tests consist of a minimal HTML page that references a JavaScript | 579 Some layout tests consist of a minimal HTML page that references a JavaScript |
581 file in `resources/`. Please do not use this pattern for new tests, as it goes | 580 file in `resources/`. Please do not use this pattern for new tests, as it goes |
582 against the minimality principle. JavaScript and CSS files should only live in | 581 against the minimality principle. JavaScript and CSS files should only live in |
583 `resources/` if they are shared by at least two test files. | 582 `resources/` if they are shared by at least two test files. |
584 *** | 583 *** |
OLD | NEW |