OLD | NEW |
1 # Writing Layout Tests | 1 # Writing Layout Tests |
2 | 2 |
3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), |
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
6 or XHTML) and lives in | 6 or XHTML) and lives in |
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
8 | 8 |
9 [TOC] | 9 [TOC] |
10 | 10 |
(...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
44 * *JavaScript Tests* are the layout test implementation of | 44 * *JavaScript Tests* are the layout test implementation of |
45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain | 45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
46 assertions written in JavaScript, and pass if the assertions evaluate to | 46 assertions written in JavaScript, and pass if the assertions evaluate to |
47 true. | 47 true. |
48 * *Reference Tests* render a test page and a reference page, and pass if the two | 48 * *Reference Tests* render a test page and a reference page, and pass if the two |
49 renderings are identical, according to a pixel-by-pixel comparison. These | 49 renderings are identical, according to a pixel-by-pixel comparison. These |
50 tests are less robust, harder to debug, and significantly slower than | 50 tests are less robust, harder to debug, and significantly slower than |
51 JavaScript tests, and are only used when JavaScript tests are insufficient, | 51 JavaScript tests, and are only used when JavaScript tests are insufficient, |
52 such as when testing paint code. | 52 such as when testing paint code. |
53 * *Pixel Tests* render a test page and compare the result against a pre-rendered | 53 * *Pixel Tests* render a test page and compare the result against a pre-rendered |
54 baseline image in the repository. Pixel tests are less robust than all | 54 baseline image in the repository. Pixel tests are less robust than the |
55 alternatives listed above, because the rendering of a page is influenced by | 55 first two types, because the rendering of a page is influenced by |
56 many factors such as the host computer's graphics card and driver, the | 56 many factors such as the host computer's graphics card and driver, the |
57 platform's text rendering system, and various user-configurable operating | 57 platform's text rendering system, and various user-configurable operating |
58 system settings. For this reason, it is common for a pixel test to have a | 58 system settings. For this reason, it is common for a pixel test to have a |
59 different reference image for each platform that Blink is tested on. Pixel | 59 different reference image for each platform that Blink is tested on, and |
60 tests are least preferred, because the reference images are | 60 the reference images are |
61 [quite cumbersome to manage](./layout_test_expectations.md). | 61 [quite cumbersome to manage](./layout_test_expectations.md). You |
62 * *Dump Render Tree (DRT) Tests* output a textual representation of the render | 62 should only write a pixel test if you cannot use a reference test. By default |
| 63 a pixel test will also dump the layout tree as text output, so they are |
| 64 similar to ... |
| 65 * *Layout tree tests*, which output a textual representation of the layout |
63 tree, which is the key data structure in Blink's page rendering system. The | 66 tree, which is the key data structure in Blink's page rendering system. The |
64 test passes if the output matches a baseline text file in the repository. In | 67 test passes if the output matches a baseline text file in the repository. |
65 addition to their text result, DRT tests can also produce an image result | 68 Layout tree tests are used as a last resort to test the internal quirks of |
66 which is compared to an image baseline, similarly to pixel tests (described | 69 the implementation, and they should be avoided in favor of one of the earlier |
67 above). A DRT test with two results (text and image) passes if _both_ results | 70 options. |
68 match the baselines in the repository. DRT tests are less desirable than all | |
69 the alternatives, because they depend on a browser implementation detail. | |
70 | |
71 ## General Principles | 71 ## General Principles |
72 | 72 |
73 The principles below are adapted from | 73 The principles below are adapted from |
74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) | 74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) |
75 and | 75 and |
76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). | 76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). |
77 | 77 |
78 *** note | 78 *** note |
79 This document intentionally uses _should_ a lot more than _must_, as defined in | 79 This document intentionally uses _should_ a lot more than _must_, as defined in |
80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a | 80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a |
(...skipping 672 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
753 If that is not possible, make the test self-describing by including a textual | 753 If that is not possible, make the test self-describing by including a textual |
754 description of the desired (passing) outcome. | 754 description of the desired (passing) outcome. |
755 * Only use the red color or the word `FAIL` to highlight errors. This does not | 755 * Only use the red color or the word `FAIL` to highlight errors. This does not |
756 apply when testing the color red. | 756 apply when testing the color red. |
757 * 🚧 Use the | 757 * 🚧 Use the |
758 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the | 758 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the |
759 variance introduced by the platform's text rendering system. This does not | 759 variance introduced by the platform's text rendering system. This does not |
760 apply when testing text, text flow, font selection, font fallback, font | 760 apply when testing text, text flow, font selection, font fallback, font |
761 features, or other typographic information. | 761 features, or other typographic information. |
762 | 762 |
| 763 TODO: Document how to opt out of generating a layout tree when generating |
| 764 pixel results. |
| 765 |
763 *** promo | 766 *** promo |
764 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result | 767 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result |
765 will always be 800x600px, because test pages are rendered in an 800x600px | 768 will always be 800x600px, because test pages are rendered in an 800x600px |
766 viewport. Pixel tests that do not specifically cover scrolling should fit in an | 769 viewport. Pixel tests that do not specifically cover scrolling should fit in an |
767 800x600px viewport without creating scrollbars. | 770 800x600px viewport without creating scrollbars. |
768 *** | 771 *** |
769 | 772 |
770 *** promo | 773 *** promo |
771 The recommendation of using Ahem in pixel tests is being discussed on | 774 The recommendation of using Ahem in pixel tests is being discussed on |
772 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). | 775 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
(...skipping 18 matching lines...) Expand all Loading... |
791 | 794 |
792 ### Tests that need to paint, raster, or draw a frame of intermediate output | 795 ### Tests that need to paint, raster, or draw a frame of intermediate output |
793 | 796 |
794 A layout test does not actually draw frames of output until the test exits. | 797 A layout test does not actually draw frames of output until the test exits. |
795 Tests that need to generate a painted frame can use | 798 Tests that need to generate a painted frame can use |
796 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a | 799 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a |
797 frame, then call the passed callback. There is also a library at | 800 frame, then call the passed callback. There is also a library at |
798 `fast/repaint/resources/text-based-repaint.js` to help with writing paint | 801 `fast/repaint/resources/text-based-repaint.js` to help with writing paint |
799 invalidation and repaint tests. | 802 invalidation and repaint tests. |
800 | 803 |
801 ## Dump Render Tree (DRT) Tests | 804 ## Layout tree tests |
802 | 805 |
803 A Dump Render Tree test renders a web page and produces up to two results, which | 806 A layout tree test renders a web page and produces up to two results, which |
804 are compared against baseline files: | 807 are compared against baseline files: |
805 | 808 |
806 * All tests output a textual representation of Blink's | 809 * All tests output a textual representation of Blink's |
807 [render tree](https://developers.google.com/web/fundamentals/performance/criti
cal-rendering-path/render-tree-construction), | 810 [layout tree](https://developers.google.com/web/fundamentals/performance/criti
cal-rendering-path/render-tree-construction) (called the render tree on that pag
e), |
808 which is compared against an `-expected.txt` text baseline. | 811 which is compared against an `-expected.txt` text baseline. |
809 * Some tests also output the image of the rendered page, which is compared | 812 * Some tests also output the image of the rendered page, which is compared |
810 against an `-expected.png` image baseline, using the same method as pixel | 813 against an `-expected.png` image baseline, using the same method as pixel |
811 tests. | 814 tests. |
812 | 815 |
813 TODO: Document the API used by DRT tests to opt out of producing image results. | 816 Whether you want a pixel test or a layout tree test depends on whether |
| 817 you care about the visual image, the details of how that image was |
| 818 constructed, or both. It is possible for multiple layout trees to produce |
| 819 the same pixel output, so it is important to make it clear in the test |
| 820 which outputs you really care about. |
814 | 821 |
815 A DRT test passes if _all_ of its results match their baselines. Like pixel | 822 TODO: Document the API used by layout tree tests to opt out of producing image |
816 tests, the output of DRT tests depends on platform-specific mechanisms, so DRT | 823 results. |
817 tests often require per-platform baselines. Furthermore, DRT tests depend on the | |
818 render tree data structure, which means that if we replace the render tree data | |
819 structure, we will need to look at each DRT test and consider whether it is | |
820 still meaningful. | |
821 | 824 |
822 For these reasons, DRT tests should **only** be used to cover aspects of the | 825 A layout tree test passes if _all_ of its results match their baselines. Like pi
xel |
823 layout code that can only be tested by looking at the render tree. Any | 826 tests, the output of layout tree tests depends on platform-specific details, |
824 combination of the other test types is preferable to a DRT test. DRT tests are | 827 so layout tree tests often require per-platform baselines. Furthermore, |
| 828 since the tests obviously depend on the layout tree structure, |
| 829 that means that if we change the layout tree you have to rebaseline each |
| 830 layout tree test to see if the results are still correct and whether the test |
| 831 is still meaningful. There are actually many cases where the layout tree |
| 832 output is misstated (i.e., wrong), because people didn't want to have to update |
| 833 existing baselines and tests. This is really unfortunate and confusing. |
| 834 |
| 835 For these reasons, layout tree tests should **only** be used to cover aspects |
| 836 of the layout code that can only be tested by looking at the layout tree. Any |
| 837 combination of the other test types is preferable to a layout tree test. |
| 838 Layout tree tests are |
825 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so | 839 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so |
826 the repository may have some unfortunate examples of DRT tests. | 840 the repository may have some unfortunate examples of layout tree tests. |
827 | 841 |
828 The following page is an example of a DRT test. | 842 |
| 843 The following page is an example of a layout tree test. |
829 | 844 |
830 ```html | 845 ```html |
831 <!doctype html> | 846 <!doctype html> |
832 <meta charset="utf-8"> | 847 <meta charset="utf-8"> |
833 <style> | 848 <style> |
834 body { font: 10px Ahem; } | 849 body { font: 10px Ahem; } |
835 span::after { | 850 span::after { |
836 content: "pass"; | 851 content: "pass"; |
837 color: green; | 852 color: green; |
838 } | 853 } |
(...skipping 20 matching lines...) Expand all Loading... |
859 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] | 874 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] |
860 LayoutTextFragment (anonymous) at (430,0) size 40x10 | 875 LayoutTextFragment (anonymous) at (430,0) size 40x10 |
861 text run at (430,0) width 40: "pass" | 876 text run at (430,0) width 40: "pass" |
862 ``` | 877 ``` |
863 | 878 |
864 Notice that the test result above depends on the size of the `<p>` text. The | 879 Notice that the test result above depends on the size of the `<p>` text. The |
865 test page uses the Ahem font (introduced above), whose main design goal is | 880 test page uses the Ahem font (introduced above), whose main design goal is |
866 consistent cross-platform rendering. Had the test used another font, its text | 881 consistent cross-platform rendering. Had the test used another font, its text |
867 baseline would have depended on the fonts installed on the testing computer, and | 882 baseline would have depended on the fonts installed on the testing computer, and |
868 on the platform's font rendering system. Please follow the pixel tests | 883 on the platform's font rendering system. Please follow the pixel tests |
869 guidelines and write reliable DRT tests! | 884 guidelines and write reliable layout tree tests! |
870 | 885 |
871 WebKit's render tree is described in | 886 WebKit's layout tree is described in |
872 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) | 887 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) |
873 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. | 888 on WebKit's blog. Some of the concepts there still apply to Blink's layout tree. |
874 | 889 |
875 ## Directory Structure | 890 ## Directory Structure |
876 | 891 |
877 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently | 892 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently |
878 lacks a strict, formal structure. The following directories have special | 893 lacks a strict, formal structure. The following directories have special |
879 meaning: | 894 meaning: |
880 | 895 |
881 * The `http/` directory hosts tests that require an HTTP server (see above). | 896 * The `http/` directory hosts tests that require an HTTP server (see above). |
882 * The `resources/` subdirectory in every directory contains binary files, such | 897 * The `resources/` subdirectory in every directory contains binary files, such |
883 as media files, and code that is shared by multiple test files. | 898 as media files, and code that is shared by multiple test files. |
884 | 899 |
885 *** note | 900 *** note |
886 Some layout tests consist of a minimal HTML page that references a JavaScript | 901 Some layout tests consist of a minimal HTML page that references a JavaScript |
887 file in `resources/`. Please do not use this pattern for new tests, as it goes | 902 file in `resources/`. Please do not use this pattern for new tests, as it goes |
888 against the minimality principle. JavaScript and CSS files should only live in | 903 against the minimality principle. JavaScript and CSS files should only live in |
889 `resources/` if they are shared by at least two test files. | 904 `resources/` if they are shared by at least two test files. |
890 *** | 905 *** |
OLD | NEW |