OLD | NEW |
---|---|
1 # Writing Layout Tests | 1 # Writing Layout Tests |
2 | 2 |
3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), |
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
6 or XHTML) and lives in | 6 or XHTML) and lives in |
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
8 | 8 |
9 [TOC] | 9 [TOC] |
10 | 10 |
(...skipping 31 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
42 * *JavaScript Tests* are the layout test implementation of | 42 * *JavaScript Tests* are the layout test implementation of |
43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain | 43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
44 assertions written in JavaScript, and pass if the assertions evaluate to | 44 assertions written in JavaScript, and pass if the assertions evaluate to |
45 true. | 45 true. |
46 * *Reference Tests* render a test page and a reference page, and pass if the two | 46 * *Reference Tests* render a test page and a reference page, and pass if the two |
47 renderings are identical, according to a pixel-by-pixel comparison. These | 47 renderings are identical, according to a pixel-by-pixel comparison. These |
48 tests are less robust, harder to debug, and significantly slower than | 48 tests are less robust, harder to debug, and significantly slower than |
49 JavaScript tests, and are only used when JavaScript tests are insufficient, | 49 JavaScript tests, and are only used when JavaScript tests are insufficient, |
50 such as when testing paint code. | 50 such as when testing paint code. |
51 * *Pixel Tests* render a test page and compare the result against a pre-rendered | 51 * *Pixel Tests* render a test page and compare the result against a pre-rendered |
52 baseline image in the repository. Pixel tests are less robust than all | 52 baseline image in the repository. Pixel tests are less robust than the |
53 alternatives listed above, because the rendering of a page is influenced by | 53 first two types, because the rendering of a page is influenced by |
54 many factors such as the host computer's graphics card and driver, the | 54 many factors such as the host computer's graphics card and driver, the |
55 platform's text rendering system, and various user-configurable operating | 55 platform's text rendering system, and various user-configurable operating |
56 system settings. For this reason, it is common for a pixel test to have a | 56 system settings. For this reason, it is common for a pixel test to have a |
57 different reference image for each platform that Blink is tested on. Pixel | 57 different reference image for each platform that Blink is tested on, and |
58 tests are least preferred, because the reference images are | 58 the reference images are |
59 [quite cumbersome to manage](./layout_test_expectations.md). | 59 [quite cumbersome to manage](./layout_test_expectations.md). You |
60 * *Dump Render Tree (DRT) Tests* output a textual representation of the render | 60 should only write a pixel test if you cannot use a reference test. By default |
61 a pixel test will also dump the render tree as text output, so they are | |
62 similar to ... | |
63 * *Render tree tests*, which output a textual representation of the render | |
cbiesinger
2016/12/02 22:15:22
We call it Layout Tree now, fwiw
| |
61 tree, which is the key data structure in Blink's page rendering system. The | 64 tree, which is the key data structure in Blink's page rendering system. The |
62 test passes if the output matches a baseline text file in the repository. In | 65 test passes if the output matches a baseline text file in the repository. |
63 addition to their text result, DRT tests can also produce an image result | 66 Render tree tests are used as a last resort to test the internal quirks of |
64 which is compared to an image baseline, similarly to pixel tests (described | 67 the implementation, and they should be avoided in favor of one of the earlier |
65 above). A DRT test with two results (text and image) passes if _both_ results | 68 options. |
66 match the baselines in the repository. DRT tests are less desirable than all | |
67 the alternatives, because they depend on a browser implementation detail. | |
68 | |
69 ## General Principles | 69 ## General Principles |
70 | 70 |
71 The principles below are adapted from | 71 The principles below are adapted from |
72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) | 72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) |
73 and | 73 and |
74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). | 74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). |
75 | 75 |
76 * Tests should be **concise**, without compromising on the principles below. | 76 * Tests should be **concise**, without compromising on the principles below. |
77 Every element and piece of code on the page should be necessary and relevant | 77 Every element and piece of code on the page should be necessary and relevant |
78 to what is being tested. For example, don't build a fully functional signup | 78 to what is being tested. For example, don't build a fully functional signup |
(...skipping 609 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
688 | 688 |
689 ### Tests that need to paint, raster, or draw a frame of intermediate output | 689 ### Tests that need to paint, raster, or draw a frame of intermediate output |
690 | 690 |
691 A layout test does not actually draw frames of output until the test exits. | 691 A layout test does not actually draw frames of output until the test exits. |
692 Tests that need to generate a painted frame can use | 692 Tests that need to generate a painted frame can use |
693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a | 693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a |
694 frame, then call the passed callback. There is also a library at | 694 frame, then call the passed callback. There is also a library at |
695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint | 695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint |
696 invalidation and repaint tests. | 696 invalidation and repaint tests. |
697 | 697 |
698 ## Dump Render Tree (DRT) Tests | 698 ## Render tree Tests |
699 | 699 |
700 A Dump Render Tree test renders a web page and produces up to two results, which | 700 A render tree test renders a web page and produces up to two results, which |
701 are compared against baseline files: | 701 are compared against baseline files: |
702 | 702 |
703 * All tests output a textual representation of Blink's | 703 * All tests output a textual representation of Blink's |
704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), | 704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), |
705 which is compared against an `-expected.txt` text baseline. | 705 which is compared against an `-expected.txt` text baseline. |
706 * Some tests also output the image of the rendered page, which is compared | 706 * Some tests also output the image of the rendered page, which is compared |
707 against an `-expected.png` image baseline, using the same method as pixel | 707 against an `-expected.png` image baseline, using the same method as pixel |
708 tests. | 708 tests. |
709 | 709 |
710 TODO: Document the API used by DRT tests to opt out of producing image results. | 710 Whether you want a pixel test or a render tree test depends on whether |
711 you care about the visual image, the details of how that image was | |
712 constructed, or both. It is possible for multiple render trees to produce | |
713 the same pixel output, so it is important to make it clear in the test | |
714 which outputs you really care about. | |
711 | 715 |
712 A DRT test passes if _all_ of its results match their baselines. Like pixel | 716 TODO: Document the API used by render tree tests to opt out of producing image |
713 tests, the output of DRT tests depends on platform-specific mechanisms, so DRT | 717 results. |
m_gsnedders
2016/12/02 14:16:44
Given the intro above says that by default "a pixe
| |
714 tests often require per-platform baselines. Furthermore, DRT tests depend on the | |
715 render tree data structure, which means that if we replace the render tree data | |
716 structure, we will need to look at each DRT test and consider whether it is | |
717 still meaningful. | |
718 | 718 |
719 For these reasons, DRT tests should **only** be used to cover aspects of the | 719 A render tree test passes if _all_ of its results match their baselines. Like pi xel |
720 layout code that can only be tested by looking at the render tree. Any | 720 tests, the output of render tree tests depends on platform-specific details, |
721 combination of the other test types is preferable to a DRT test. DRT tests are | 721 so render tree tests often require per-platform baselines. Furthermore, |
722 since the tests obviously depend on the render tree structure, | |
723 that means that if we change the render tree you have to rebaseline each | |
724 render tree test to see if the results are still correct and whether the test | |
725 is still meaningful. There are actually many cases where the render tree | |
pwnall
2016/12/02 03:20:48
Could "many cases" link to a crbug that tracks rem
| |
726 output is misstated (i.e., wrong), because people didn't want to have to update | |
727 existing baselines and tests. This is really unfortunate and confusing. | |
728 | |
729 For these reasons, render tree tests should **only** be used to cover aspects | |
730 of the layout code that can only be tested by looking at the render tree. Any | |
731 combination of the other test types is preferable to a render tree test. | |
732 render tree tests are | |
m_gsnedders
2016/12/02 14:16:44
(grammar): needs a capital R
| |
722 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so | 733 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so |
723 the repository may have some unfortunate examples of DRT tests. | 734 the repository may have some unfortunate examples of render tree tests. |
724 | 735 |
725 The following page is an example of a DRT test. | 736 |
737 The following page is an example of a render tree test. | |
726 | 738 |
727 ```html | 739 ```html |
728 <!doctype html> | 740 <!doctype html> |
729 <meta charset="utf-8"> | 741 <meta charset="utf-8"> |
730 <style> | 742 <style> |
731 body { font: 10px Ahem; } | 743 body { font: 10px Ahem; } |
732 span::after { | 744 span::after { |
733 content: "pass"; | 745 content: "pass"; |
734 color: green; | 746 color: green; |
735 } | 747 } |
(...skipping 20 matching lines...) Expand all Loading... | |
756 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] | 768 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] |
757 LayoutTextFragment (anonymous) at (430,0) size 40x10 | 769 LayoutTextFragment (anonymous) at (430,0) size 40x10 |
758 text run at (430,0) width 40: "pass" | 770 text run at (430,0) width 40: "pass" |
759 ``` | 771 ``` |
760 | 772 |
761 Notice that the test result above depends on the size of the `<p>` text. The | 773 Notice that the test result above depends on the size of the `<p>` text. The |
762 test page uses the Ahem font (introduced above), whose main design goal is | 774 test page uses the Ahem font (introduced above), whose main design goal is |
763 consistent cross-platform rendering. Had the test used another font, its text | 775 consistent cross-platform rendering. Had the test used another font, its text |
764 baseline would have depended on the fonts installed on the testing computer, and | 776 baseline would have depended on the fonts installed on the testing computer, and |
765 on the platform's font rendering system. Please follow the pixel tests | 777 on the platform's font rendering system. Please follow the pixel tests |
766 guidelines and write reliable DRT tests! | 778 guidelines and write reliable render tree tests! |
767 | 779 |
768 WebKit's render tree is described in | 780 WebKit's render tree is described in |
769 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) | 781 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) |
770 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. | 782 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. |
771 | 783 |
772 ## Directory Structure | 784 ## Directory Structure |
773 | 785 |
774 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently | 786 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently |
775 lacks a strict, formal structure. The following directories have special | 787 lacks a strict, formal structure. The following directories have special |
776 meaning: | 788 meaning: |
777 | 789 |
778 * The `http/` directory hosts tests that require an HTTP server (see above). | 790 * The `http/` directory hosts tests that require an HTTP server (see above). |
779 * The `resources/` subdirectory in every directory contains binary files, such | 791 * The `resources/` subdirectory in every directory contains binary files, such |
780 as media files, and code that is shared by multiple test files. | 792 as media files, and code that is shared by multiple test files. |
781 | 793 |
782 *** note | 794 *** note |
783 Some layout tests consist of a minimal HTML page that references a JavaScript | 795 Some layout tests consist of a minimal HTML page that references a JavaScript |
784 file in `resources/`. Please do not use this pattern for new tests, as it goes | 796 file in `resources/`. Please do not use this pattern for new tests, as it goes |
785 against the minimality principle. JavaScript and CSS files should only live in | 797 against the minimality principle. JavaScript and CSS files should only live in |
786 `resources/` if they are shared by at least two test files. | 798 `resources/` if they are shared by at least two test files. |
787 *** | 799 *** |
OLD | NEW |