Chromium Code Reviews| OLD | NEW |
|---|---|
| 1 # Writing Layout Tests | 1 # Writing Layout Tests |
| 2 | 2 |
| 3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
| 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), |
| 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
| 6 or XHTML) and lives in | 6 or XHTML) and lives in |
| 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
| 8 | 8 |
| 9 [TOC] | 9 [TOC] |
| 10 | 10 |
| 11 ## Overview | 11 ## Overview |
| 12 | 12 |
| 13 Layout tests should be used to accomplish one of the following goals: | 13 Layout tests should be used to accomplish one of the following goals: |
| 14 | 14 |
| 15 1. The entire surface of Blink that is exposed to the Web should be covered by | 15 1. The entire surface of Blink that is exposed to the Web should be covered by |
| 16 tests that we contribute to the | 16 tests that we contribute to [web-platform-tests](./web_platform_tests.md) |
| 17 [Web Platform Tests Project](./web_platform_tests.md) (WPT). This helps us | 17 (WPT). This helps us avoid regressions, and helps us identify Web Platform |
|
pwnall
2017/02/16 19:10:40
do we still use WPT, or should we use wpt for web-
foolip
2017/02/16 19:18:58
I have consulted my inbox and see people using bot
| |
| 18 avoid regressions, and helps us identify Web Platform areas where the major | 18 areas where the major browsers don't have interoperable implementations. |
| 19 browsers don't have interoperable implementations. Furthermore, by | 19 Furthermore, by contributing to projects such as WPT, we share the burden of |
| 20 contributing to projects such as WPT, we share the burden of writing tests | 20 writing tests with the other browser vendors, and we help all the browsers |
| 21 with the other browser vendors, and we help all the browsers get better. This | 21 get better. This is very much in line with our goal to move the Web forward. |
| 22 is very much in line with our goal to move the Web forward. | |
| 23 2. When a Blink feature cannot be tested using the tools provided by WPT, and | 22 2. When a Blink feature cannot be tested using the tools provided by WPT, and |
| 24 cannot be easily covered by | 23 cannot be easily covered by |
| 25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), | 24 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), |
| 26 the feature must be covered by layout tests, to avoid unexpected regressions. | 25 the feature must be covered by layout tests, to avoid unexpected regressions. |
| 27 These tests will use Blink-specific testing APIs that are only available in | 26 These tests will use Blink-specific testing APIs that are only available in |
| 28 [content_shell](./layout_tests_in_content_shell.md). | 27 [content_shell](./layout_tests_in_content_shell.md). |
| 29 | 28 |
| 30 *** promo | 29 *** promo |
| 31 If you know that Blink layout tests are upstreamed to other projects, such as | 30 If you know that Blink layout tests are upstreamed to other projects, such as |
| 32 [test262](https://github.com/tc39/test262), please update this document. Most | 31 [test262](https://github.com/tc39/test262), please update this document. Most |
| (...skipping 34 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 67 test passes if the output matches a baseline text file in the repository. | 66 test passes if the output matches a baseline text file in the repository. |
| 68 Layout tree tests are used as a last resort to test the internal quirks of | 67 Layout tree tests are used as a last resort to test the internal quirks of |
| 69 the implementation, and they should be avoided in favor of one of the earlier | 68 the implementation, and they should be avoided in favor of one of the earlier |
| 70 options. | 69 options. |
| 71 | 70 |
| 72 ## General Principles | 71 ## General Principles |
| 73 | 72 |
| 74 | 73 |
| 75 Tests should be written under the assumption that they will be upstreamed | 74 Tests should be written under the assumption that they will be upstreamed |
| 76 to the WPT project. To this end, tests should follow the | 75 to the WPT project. To this end, tests should follow the |
| 77 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). | 76 [WPT guidelines](http://web-platform-tests.org/writing-tests/). |
| 78 | 77 |
| 79 | 78 |
| 80 There is no style guide that applies to all layout tests. However, some projects | 79 There is no style guide that applies to all layout tests. However, some projects |
| 81 have adopted style guides, such as the | 80 have adopted style guides, such as the |
| 82 [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/t esting). | 81 [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/t esting). |
| 83 | 82 |
| 84 Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most | 83 Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most |
| 85 important WPT guidelines and highlights some JavaScript concepts that are worth | 84 important WPT guidelines and highlights some JavaScript concepts that are worth |
| 86 paying attention to when trying to infer style rules from existing tests. If | 85 paying attention to when trying to infer style rules from existing tests. If |
| 87 you're unopinionated and looking for a style guide to follow, the document also | 86 you're unopinionated and looking for a style guide to follow, the document also |
| 88 suggests some defaults. | 87 suggests some defaults. |
| 89 | 88 |
| 90 ## JavaScript Tests | 89 ## JavaScript Tests |
| 91 | 90 |
| 92 Whenever possible, the testing criteria should be expressed in JavaScript. The | 91 Whenever possible, the testing criteria should be expressed in JavaScript. The |
| 93 alternatives, which will be described in future sections, result in slower and | 92 alternatives, which will be described in future sections, result in slower and |
| 94 less reliable tests. | 93 less reliable tests. |
| 95 | 94 |
| 96 All new JavaScript tests should be written using the | 95 All new JavaScript tests should be written using the |
| 97 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This | 96 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This |
| 98 framework is used by the tests in the | 97 framework is used by the tests in the |
| 99 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, | 98 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, |
| 100 which is shared with all the other browser vendors, so `testharness.js` tests | 99 which is shared with all the other browser vendors, so `testharness.js` tests |
| 101 are more accessible to browser developers. | 100 are more accessible to browser developers. |
| 102 | 101 |
| 103 As a shared framework, `testharness.js` enjoys high-quality documentation, such | 102 See the [API documentation](https://github.com/w3c/testharness.js/blob/master/do cs/api.md) |
| 104 as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and | 103 for a thorough introduction to `testharness.js`. |
| 105 [API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.m d). | 104 |
| 106 Layout tests should follow the recommendations of the above documents. | 105 Layout tests should follow the recommendations of the above documentation. |
| 107 Furthermore, layout tests should include relevant | 106 Furthermore, layout tests should include relevant |
| 108 [metadata](http://testthewebforward.org/docs/css-metadata.html). The | 107 [metadata](http://web-platform-tests.org/writing-tests/css-metadata.html). The |
| 109 specification URL (in `<link rel="help">`) is almost always relevant, and is | 108 specification URL (in `<link rel="help">`) is almost always relevant, and is |
| 110 incredibly helpful to a developer who needs to understand the test quickly. | 109 incredibly helpful to a developer who needs to understand the test quickly. |
| 111 | 110 |
| 112 Below is a skeleton for a JavaScript test embedded in an HTML page. Note that, | 111 Below is a skeleton for a JavaScript test embedded in an HTML page. Note that, |
| 113 in order to follow the minimality guideline, the test omits the tags `<html>`, | 112 in order to follow the minimality guideline, the test omits the tags `<html>`, |
| 114 `<head>`, and `<body>`, as they can be inferred by the HTML parser. | 113 `<head>`, and `<body>`, as they can be inferred by the HTML parser. |
| 115 | 114 |
| 116 ```html | 115 ```html |
| 117 <!doctype html> | 116 <!doctype html> |
| 118 <title>JavaScript: the true literal is immutable and equal to itself</title> | 117 <title>JavaScript: the true literal is immutable and equal to itself</title> |
| (...skipping 227 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 346 Reference tests, also known as reftests, perform a pixel-by-pixel comparison | 345 Reference tests, also known as reftests, perform a pixel-by-pixel comparison |
| 347 between the rendered image of a test page and the rendered image of a reference | 346 between the rendered image of a test page and the rendered image of a reference |
| 348 page. Most reference tests pass if the two images match, but there are cases | 347 page. Most reference tests pass if the two images match, but there are cases |
| 349 where it is useful to have a test pass when the two images do _not_ match. | 348 where it is useful to have a test pass when the two images do _not_ match. |
| 350 | 349 |
| 351 Reference tests are more difficult to debug than JavaScript tests, and tend to | 350 Reference tests are more difficult to debug than JavaScript tests, and tend to |
| 352 be slower as well. Therefore, they should only be used for functionality that | 351 be slower as well. Therefore, they should only be used for functionality that |
| 353 cannot be covered by JavaScript tests. | 352 cannot be covered by JavaScript tests. |
| 354 | 353 |
| 355 New reference tests should follow the | 354 New reference tests should follow the |
| 356 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The | 355 [WPT reftests guidelines](http://web-platform-tests.org/writing-tests/reftests.h tml). |
| 357 most important points are summarized below. | 356 The most important points are summarized below. |
| 358 | 357 |
| 359 * 🚧 The test page declares the reference page using a | 358 * 🚧 The test page declares the reference page using a |
| 360 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test | 359 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test |
| 361 passes when the test image matches or does not match the reference image. | 360 passes when the test image matches or does not match the reference image. |
| 362 * The reference page must not use the feature being tested. Otherwise, the test | 361 * The reference page must not use the feature being tested. Otherwise, the test |
| 363 is meaningless. | 362 is meaningless. |
| 364 * The reference page should be as simple as possible, and should not depend on | 363 * The reference page should be as simple as possible, and should not depend on |
| 365 advanced features. Ideally, the reference page should render as intended even | 364 advanced features. Ideally, the reference page should render as intended even |
| 366 on browsers with poor CSS support. | 365 on browsers with poor CSS support. |
| 367 * Reference tests should be self-describing. | 366 * Reference tests should be self-describing. |
| (...skipping 51 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 419 `window.testRunner.dumpDragImage()` create an image result that is associated | 418 `window.testRunner.dumpDragImage()` create an image result that is associated |
| 420 with the test. The image result is compared against an image baseline, which is | 419 with the test. The image result is compared against an image baseline, which is |
| 421 an `-expected.png` file associated with the test, and the test passes if the | 420 an `-expected.png` file associated with the test, and the test passes if the |
| 422 image result is identical to the baseline, according to a pixel-by-pixel | 421 image result is identical to the baseline, according to a pixel-by-pixel |
| 423 comparison. Tests that have image results (and baselines) are called **pixel | 422 comparison. Tests that have image results (and baselines) are called **pixel |
| 424 tests**. | 423 tests**. |
| 425 | 424 |
| 426 Pixel tests should still follow the principles laid out above. Pixel tests pose | 425 Pixel tests should still follow the principles laid out above. Pixel tests pose |
| 427 unique challenges to the desire to have *self-describing* and *cross-platform* | 426 unique challenges to the desire to have *self-describing* and *cross-platform* |
| 428 tests. The | 427 tests. The |
| 429 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli nes.html) | 428 [WPT rendering test guidelines](http://web-platform-tests.org/writing-tests/rend ering.html) |
| 430 contain useful guidance. The most relevant pieces of advice are below. | 429 contain useful guidance. The most relevant pieces of advice are below. |
| 431 | 430 |
| 432 * Whenever possible, use a green paragraph / page / square to indicate success. | 431 * Whenever possible, use a green paragraph / page / square to indicate success. |
| 433 If that is not possible, make the test self-describing by including a textual | 432 If that is not possible, make the test self-describing by including a textual |
| 434 description of the desired (passing) outcome. | 433 description of the desired (passing) outcome. |
| 435 * Only use the red color or the word `FAIL` to highlight errors. This does not | 434 * Only use the red color or the word `FAIL` to highlight errors. This does not |
| 436 apply when testing the color red. | 435 apply when testing the color red. |
| 437 * 🚧 Use the | 436 * 🚧 Use the |
| 438 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the | 437 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the |
| 439 variance introduced by the platform's text rendering system. This does not | 438 variance introduced by the platform's text rendering system. This does not |
| (...skipping 135 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 575 * The `http/` directory hosts tests that require an HTTP server (see above). | 574 * The `http/` directory hosts tests that require an HTTP server (see above). |
| 576 * The `resources/` subdirectory in every directory contains binary files, such | 575 * The `resources/` subdirectory in every directory contains binary files, such |
| 577 as media files, and code that is shared by multiple test files. | 576 as media files, and code that is shared by multiple test files. |
| 578 | 577 |
| 579 *** note | 578 *** note |
| 580 Some layout tests consist of a minimal HTML page that references a JavaScript | 579 Some layout tests consist of a minimal HTML page that references a JavaScript |
| 581 file in `resources/`. Please do not use this pattern for new tests, as it goes | 580 file in `resources/`. Please do not use this pattern for new tests, as it goes |
| 582 against the minimality principle. JavaScript and CSS files should only live in | 581 against the minimality principle. JavaScript and CSS files should only live in |
| 583 `resources/` if they are shared by at least two test files. | 582 `resources/` if they are shared by at least two test files. |
| 584 *** | 583 *** |
| OLD | NEW |