OLD | NEW |
---|---|
1 # Writing Layout Tests | 1 # Writing Layout Tests |
2 | 2 |
3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), |
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
6 or XHTML) and lives in | 6 or XHTML) and lives in |
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
8 | 8 |
9 [TOC] | 9 [TOC] |
10 | 10 |
(...skipping 31 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
42 * *JavaScript Tests* are the layout test implementation of | 42 * *JavaScript Tests* are the layout test implementation of |
43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain | 43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
44 assertions written in JavaScript, and pass if the assertions evaluate to | 44 assertions written in JavaScript, and pass if the assertions evaluate to |
45 true. | 45 true. |
46 * *Reference Tests* render a test page and a reference page, and pass if the two | 46 * *Reference Tests* render a test page and a reference page, and pass if the two |
47 renderings are identical, according to a pixel-by-pixel comparison. These | 47 renderings are identical, according to a pixel-by-pixel comparison. These |
48 tests are less robust, harder to debug, and significantly slower than | 48 tests are less robust, harder to debug, and significantly slower than |
49 JavaScript tests, and are only used when JavaScript tests are insufficient, | 49 JavaScript tests, and are only used when JavaScript tests are insufficient, |
50 such as when testing paint code. | 50 such as when testing paint code. |
51 * *Pixel Tests* render a test page and compare the result against a pre-rendered | 51 * *Pixel Tests* render a test page and compare the result against a pre-rendered |
52 baseline image in the repository. Pixel tests are less robust than all | 52 baseline image in the repository. Pixel tests are less robust than the |
53 alternatives listed above, because the rendering of a page is influenced by | 53 first two types, because the rendering of a page is influenced by |
pwnall
2016/12/02 02:35:10
Thanks for catching this inconsistency!
| |
54 many factors such as the host computer's graphics card and driver, the | 54 many factors such as the host computer's graphics card and driver, the |
55 platform's text rendering system, and various user-configurable operating | 55 platform's text rendering system, and various user-configurable operating |
56 system settings. For this reason, it is common for a pixel test to have a | 56 system settings. For this reason, it is common for a pixel test to have a |
57 different reference image for each platform that Blink is tested on. Pixel | 57 different reference image for each platform that Blink is tested on, and |
58 tests are least preferred, because the reference images are | 58 the reference images are |
59 [quite cumbersome to manage](./layout_test_expectations.md). | 59 [quite cumbersome to manage](./layout_test_expectations.md). You |
60 * *Dump Render Tree (DRT) Tests* output a textual representation of the render | 60 should only write a pixel test if you cannot use a reference test. By default |
61 a pixel test will also dump the render tree as text output, so they are | |
62 similar to ... | |
63 * *Render tree tests*, which output a textual representation of the render | |
61 tree, which is the key data structure in Blink's page rendering system. The | 64 tree, which is the key data structure in Blink's page rendering system. The |
62 test passes if the output matches a baseline text file in the repository. In | 65 test passes if the output matches a baseline text file in the repository. |
63 addition to their text result, DRT tests can also produce an image result | 66 Render tree tests are used as a last resort to test the internal quirks of |
64 which is compared to an image baseline, similarly to pixel tests (described | 67 the implementation, and they should be avoided in favor of one of the earlier |
65 above). A DRT test with two results (text and image) passes if _both_ results | 68 options. Most render tree tests also produce image results, and so these |
pwnall
2016/12/02 02:35:10
I think that the text starting at "Most render tre
Dirk Pranke
2016/12/02 02:44:54
Okay.
| |
66 match the baselines in the repository. DRT tests are less desirable than all | 69 tests are easily confused with Pixel tests. The distinction lies in whether |
67 the alternatives, because they depend on a browser implementation detail. | 70 you care about the visual image, the details of how that image was |
71 constructed, or both. It is possible for multiple render trees to produce | |
72 the same pixel output, so it is important to make it clear in the test | |
73 which outputs you really care about. | |
68 | 74 |
69 ## General Principles | 75 ## General Principles |
70 | 76 |
71 The principles below are adapted from | 77 The principles below are adapted from |
72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) | 78 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) |
73 and | 79 and |
74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). | 80 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). |
75 | 81 |
76 * Tests should be **concise**, without compromising on the principles below. | 82 * Tests should be **concise**, without compromising on the principles below. |
77 Every element and piece of code on the page should be necessary and relevant | 83 Every element and piece of code on the page should be necessary and relevant |
(...skipping 610 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
688 | 694 |
689 ### Tests that need to paint, raster, or draw a frame of intermediate output | 695 ### Tests that need to paint, raster, or draw a frame of intermediate output |
690 | 696 |
691 A layout test does not actually draw frames of output until the test exits. | 697 A layout test does not actually draw frames of output until the test exits. |
692 Tests that need to generate a painted frame can use | 698 Tests that need to generate a painted frame can use |
693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a | 699 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a |
694 frame, then call the passed callback. There is also a library at | 700 frame, then call the passed callback. There is also a library at |
695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint | 701 `fast/repaint/resources/text-based-repaint.js` to help with writing paint |
696 invalidation and repaint tests. | 702 invalidation and repaint tests. |
697 | 703 |
698 ## Dump Render Tree (DRT) Tests | 704 ## Dump Render Tree (DRT) Tests |
pwnall
2016/12/02 02:35:10
Can you please make the name here consistent with
Dirk Pranke
2016/12/02 02:44:54
Will do. It didn't even occur to me to check for m
| |
699 | 705 |
700 A Dump Render Tree test renders a web page and produces up to two results, which | 706 A Dump Render Tree test renders a web page and produces up to two results, which |
701 are compared against baseline files: | 707 are compared against baseline files: |
702 | 708 |
703 * All tests output a textual representation of Blink's | 709 * All tests output a textual representation of Blink's |
704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), | 710 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), |
705 which is compared against an `-expected.txt` text baseline. | 711 which is compared against an `-expected.txt` text baseline. |
706 * Some tests also output the image of the rendered page, which is compared | 712 * Some tests also output the image of the rendered page, which is compared |
707 against an `-expected.png` image baseline, using the same method as pixel | 713 against an `-expected.png` image baseline, using the same method as pixel |
708 tests. | 714 tests. |
(...skipping 69 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
778 * The `http/` directory hosts tests that require an HTTP server (see above). | 784 * The `http/` directory hosts tests that require an HTTP server (see above). |
779 * The `resources/` subdirectory in every directory contains binary files, such | 785 * The `resources/` subdirectory in every directory contains binary files, such |
780 as media files, and code that is shared by multiple test files. | 786 as media files, and code that is shared by multiple test files. |
781 | 787 |
782 *** note | 788 *** note |
783 Some layout tests consist of a minimal HTML page that references a JavaScript | 789 Some layout tests consist of a minimal HTML page that references a JavaScript |
784 file in `resources/`. Please do not use this pattern for new tests, as it goes | 790 file in `resources/`. Please do not use this pattern for new tests, as it goes |
785 against the minimality principle. JavaScript and CSS files should only live in | 791 against the minimality principle. JavaScript and CSS files should only live in |
786 `resources/` if they are shared by at least two test files. | 792 `resources/` if they are shared by at least two test files. |
787 *** | 793 *** |
OLD | NEW |