Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(191)

Side by Side Diff: docs/testing/writing_layout_tests.md

Issue 2696323003: Link to new web-platform-tests.org documentation from our own (Closed)
Patch Set: Created 3 years, 10 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « docs/testing/web_platform_tests.md ('k') | third_party/WebKit/Source/core/dom/custom/README.md » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Writing Layout Tests 1 # Writing Layout Tests
2 2
3 _Layout tests_ is a bit of a misnomer. This term is 3 _Layout tests_ is a bit of a misnomer. This term is
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /),
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, 5 and we use it to refer to every test that is written as a Web page (HTML, SVG,
6 or XHTML) and lives in 6 or XHTML) and lives in
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8 8
9 [TOC] 9 [TOC]
10 10
11 ## Overview 11 ## Overview
12 12
13 Layout tests should be used to accomplish one of the following goals: 13 Layout tests should be used to accomplish one of the following goals:
14 14
15 1. The entire surface of Blink that is exposed to the Web should be covered by 15 1. The entire surface of Blink that is exposed to the Web should be covered by
16 tests that we contribute to the 16 tests that we contribute to [web-platform-tests](./web_platform_tests.md)
17 [Web Platform Tests Project](./web_platform_tests.md) (WPT). This helps us 17 (WPT). This helps us avoid regressions, and helps us identify Web Platform
pwnall 2017/02/16 19:10:40 do we still use WPT, or should we use wpt for web-
foolip 2017/02/16 19:18:58 I have consulted my inbox and see people using bot
18 avoid regressions, and helps us identify Web Platform areas where the major 18 areas where the major browsers don't have interoperable implementations.
19 browsers don't have interoperable implementations. Furthermore, by 19 Furthermore, by contributing to projects such as WPT, we share the burden of
20 contributing to projects such as WPT, we share the burden of writing tests 20 writing tests with the other browser vendors, and we help all the browsers
21 with the other browser vendors, and we help all the browsers get better. This 21 get better. This is very much in line with our goal to move the Web forward.
22 is very much in line with our goal to move the Web forward.
23 2. When a Blink feature cannot be tested using the tools provided by WPT, and 22 2. When a Blink feature cannot be tested using the tools provided by WPT, and
24 cannot be easily covered by 23 cannot be easily covered by
25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs), 24 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Sour ce/web/tests/?q=webframetest&sq=package:chromium&type=cs),
26 the feature must be covered by layout tests, to avoid unexpected regressions. 25 the feature must be covered by layout tests, to avoid unexpected regressions.
27 These tests will use Blink-specific testing APIs that are only available in 26 These tests will use Blink-specific testing APIs that are only available in
28 [content_shell](./layout_tests_in_content_shell.md). 27 [content_shell](./layout_tests_in_content_shell.md).
29 28
30 *** promo 29 *** promo
31 If you know that Blink layout tests are upstreamed to other projects, such as 30 If you know that Blink layout tests are upstreamed to other projects, such as
32 [test262](https://github.com/tc39/test262), please update this document. Most 31 [test262](https://github.com/tc39/test262), please update this document. Most
(...skipping 34 matching lines...) Expand 10 before | Expand all | Expand 10 after
67 test passes if the output matches a baseline text file in the repository. 66 test passes if the output matches a baseline text file in the repository.
68 Layout tree tests are used as a last resort to test the internal quirks of 67 Layout tree tests are used as a last resort to test the internal quirks of
69 the implementation, and they should be avoided in favor of one of the earlier 68 the implementation, and they should be avoided in favor of one of the earlier
70 options. 69 options.
71 70
72 ## General Principles 71 ## General Principles
73 72
74 73
75 Tests should be written under the assumption that they will be upstreamed 74 Tests should be written under the assumption that they will be upstreamed
76 to the WPT project. To this end, tests should follow the 75 to the WPT project. To this end, tests should follow the
77 [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). 76 [WPT guidelines](http://web-platform-tests.org/writing-tests/).
78 77
79 78
80 There is no style guide that applies to all layout tests. However, some projects 79 There is no style guide that applies to all layout tests. However, some projects
81 have adopted style guides, such as the 80 have adopted style guides, such as the
82 [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/t esting). 81 [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/t esting).
83 82
84 Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most 83 Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
85 important WPT guidelines and highlights some JavaScript concepts that are worth 84 important WPT guidelines and highlights some JavaScript concepts that are worth
86 paying attention to when trying to infer style rules from existing tests. If 85 paying attention to when trying to infer style rules from existing tests. If
87 you're unopinionated and looking for a style guide to follow, the document also 86 you're unopinionated and looking for a style guide to follow, the document also
88 suggests some defaults. 87 suggests some defaults.
89 88
90 ## JavaScript Tests 89 ## JavaScript Tests
91 90
92 Whenever possible, the testing criteria should be expressed in JavaScript. The 91 Whenever possible, the testing criteria should be expressed in JavaScript. The
93 alternatives, which will be described in future sections, result in slower and 92 alternatives, which will be described in future sections, result in slower and
94 less reliable tests. 93 less reliable tests.
95 94
96 All new JavaScript tests should be written using the 95 All new JavaScript tests should be written using the
97 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This 96 [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This
98 framework is used by the tests in the 97 framework is used by the tests in the
99 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, 98 [web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
100 which is shared with all the other browser vendors, so `testharness.js` tests 99 which is shared with all the other browser vendors, so `testharness.js` tests
101 are more accessible to browser developers. 100 are more accessible to browser developers.
102 101
103 As a shared framework, `testharness.js` enjoys high-quality documentation, such 102 See the [API documentation](https://github.com/w3c/testharness.js/blob/master/do cs/api.md)
104 as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and 103 for a thorough introduction to `testharness.js`.
105 [API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.m d). 104
106 Layout tests should follow the recommendations of the above documents. 105 Layout tests should follow the recommendations of the above documentation.
107 Furthermore, layout tests should include relevant 106 Furthermore, layout tests should include relevant
108 [metadata](http://testthewebforward.org/docs/css-metadata.html). The 107 [metadata](http://web-platform-tests.org/writing-tests/css-metadata.html). The
109 specification URL (in `<link rel="help">`) is almost always relevant, and is 108 specification URL (in `<link rel="help">`) is almost always relevant, and is
110 incredibly helpful to a developer who needs to understand the test quickly. 109 incredibly helpful to a developer who needs to understand the test quickly.
111 110
112 Below is a skeleton for a JavaScript test embedded in an HTML page. Note that, 111 Below is a skeleton for a JavaScript test embedded in an HTML page. Note that,
113 in order to follow the minimality guideline, the test omits the tags `<html>`, 112 in order to follow the minimality guideline, the test omits the tags `<html>`,
114 `<head>`, and `<body>`, as they can be inferred by the HTML parser. 113 `<head>`, and `<body>`, as they can be inferred by the HTML parser.
115 114
116 ```html 115 ```html
117 <!doctype html> 116 <!doctype html>
118 <title>JavaScript: the true literal is immutable and equal to itself</title> 117 <title>JavaScript: the true literal is immutable and equal to itself</title>
(...skipping 227 matching lines...) Expand 10 before | Expand all | Expand 10 after
346 Reference tests, also known as reftests, perform a pixel-by-pixel comparison 345 Reference tests, also known as reftests, perform a pixel-by-pixel comparison
347 between the rendered image of a test page and the rendered image of a reference 346 between the rendered image of a test page and the rendered image of a reference
348 page. Most reference tests pass if the two images match, but there are cases 347 page. Most reference tests pass if the two images match, but there are cases
349 where it is useful to have a test pass when the two images do _not_ match. 348 where it is useful to have a test pass when the two images do _not_ match.
350 349
351 Reference tests are more difficult to debug than JavaScript tests, and tend to 350 Reference tests are more difficult to debug than JavaScript tests, and tend to
352 be slower as well. Therefore, they should only be used for functionality that 351 be slower as well. Therefore, they should only be used for functionality that
353 cannot be covered by JavaScript tests. 352 cannot be covered by JavaScript tests.
354 353
355 New reference tests should follow the 354 New reference tests should follow the
356 [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The 355 [WPT reftests guidelines](http://web-platform-tests.org/writing-tests/reftests.h tml).
357 most important points are summarized below. 356 The most important points are summarized below.
358 357
359 * &#x1F6A7; The test page declares the reference page using a 358 * &#x1F6A7; The test page declares the reference page using a
360 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test 359 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test
361 passes when the test image matches or does not match the reference image. 360 passes when the test image matches or does not match the reference image.
362 * The reference page must not use the feature being tested. Otherwise, the test 361 * The reference page must not use the feature being tested. Otherwise, the test
363 is meaningless. 362 is meaningless.
364 * The reference page should be as simple as possible, and should not depend on 363 * The reference page should be as simple as possible, and should not depend on
365 advanced features. Ideally, the reference page should render as intended even 364 advanced features. Ideally, the reference page should render as intended even
366 on browsers with poor CSS support. 365 on browsers with poor CSS support.
367 * Reference tests should be self-describing. 366 * Reference tests should be self-describing.
(...skipping 51 matching lines...) Expand 10 before | Expand all | Expand 10 after
419 `window.testRunner.dumpDragImage()` create an image result that is associated 418 `window.testRunner.dumpDragImage()` create an image result that is associated
420 with the test. The image result is compared against an image baseline, which is 419 with the test. The image result is compared against an image baseline, which is
421 an `-expected.png` file associated with the test, and the test passes if the 420 an `-expected.png` file associated with the test, and the test passes if the
422 image result is identical to the baseline, according to a pixel-by-pixel 421 image result is identical to the baseline, according to a pixel-by-pixel
423 comparison. Tests that have image results (and baselines) are called **pixel 422 comparison. Tests that have image results (and baselines) are called **pixel
424 tests**. 423 tests**.
425 424
426 Pixel tests should still follow the principles laid out above. Pixel tests pose 425 Pixel tests should still follow the principles laid out above. Pixel tests pose
427 unique challenges to the desire to have *self-describing* and *cross-platform* 426 unique challenges to the desire to have *self-describing* and *cross-platform*
428 tests. The 427 tests. The
429 [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guideli nes.html) 428 [WPT rendering test guidelines](http://web-platform-tests.org/writing-tests/rend ering.html)
430 contain useful guidance. The most relevant pieces of advice are below. 429 contain useful guidance. The most relevant pieces of advice are below.
431 430
432 * Whenever possible, use a green paragraph / page / square to indicate success. 431 * Whenever possible, use a green paragraph / page / square to indicate success.
433 If that is not possible, make the test self-describing by including a textual 432 If that is not possible, make the test self-describing by including a textual
434 description of the desired (passing) outcome. 433 description of the desired (passing) outcome.
435 * Only use the red color or the word `FAIL` to highlight errors. This does not 434 * Only use the red color or the word `FAIL` to highlight errors. This does not
436 apply when testing the color red. 435 apply when testing the color red.
437 * &#x1F6A7; Use the 436 * &#x1F6A7; Use the
438 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the 437 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the
439 variance introduced by the platform's text rendering system. This does not 438 variance introduced by the platform's text rendering system. This does not
(...skipping 135 matching lines...) Expand 10 before | Expand all | Expand 10 after
575 * The `http/` directory hosts tests that require an HTTP server (see above). 574 * The `http/` directory hosts tests that require an HTTP server (see above).
576 * The `resources/` subdirectory in every directory contains binary files, such 575 * The `resources/` subdirectory in every directory contains binary files, such
577 as media files, and code that is shared by multiple test files. 576 as media files, and code that is shared by multiple test files.
578 577
579 *** note 578 *** note
580 Some layout tests consist of a minimal HTML page that references a JavaScript 579 Some layout tests consist of a minimal HTML page that references a JavaScript
581 file in `resources/`. Please do not use this pattern for new tests, as it goes 580 file in `resources/`. Please do not use this pattern for new tests, as it goes
582 against the minimality principle. JavaScript and CSS files should only live in 581 against the minimality principle. JavaScript and CSS files should only live in
583 `resources/` if they are shared by at least two test files. 582 `resources/` if they are shared by at least two test files.
584 *** 583 ***
OLDNEW
« no previous file with comments | « docs/testing/web_platform_tests.md ('k') | third_party/WebKit/Source/core/dom/custom/README.md » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698