| OLD | NEW |
| 1 # Writing Layout Tests | 1 # Writing Layout Tests |
| 2 | 2 |
| 3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
| 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory
/), |
| 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
| 6 or XHTML) and lives in | 6 or XHTML) and lives in |
| 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
| 8 | 8 |
| 9 [TOC] | 9 [TOC] |
| 10 | 10 |
| (...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 44 * *JavaScript Tests* are the layout test implementation of | 44 * *JavaScript Tests* are the layout test implementation of |
| 45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain | 45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
| 46 assertions written in JavaScript, and pass if the assertions evaluate to | 46 assertions written in JavaScript, and pass if the assertions evaluate to |
| 47 true. | 47 true. |
| 48 * *Reference Tests* render a test page and a reference page, and pass if the two | 48 * *Reference Tests* render a test page and a reference page, and pass if the two |
| 49 renderings are identical, according to a pixel-by-pixel comparison. These | 49 renderings are identical, according to a pixel-by-pixel comparison. These |
| 50 tests are less robust, harder to debug, and significantly slower than | 50 tests are less robust, harder to debug, and significantly slower than |
| 51 JavaScript tests, and are only used when JavaScript tests are insufficient, | 51 JavaScript tests, and are only used when JavaScript tests are insufficient, |
| 52 such as when testing paint code. | 52 such as when testing paint code. |
| 53 * *Pixel Tests* render a test page and compare the result against a pre-rendered | 53 * *Pixel Tests* render a test page and compare the result against a pre-rendered |
| 54 baseline image in the repository. Pixel tests are less robust than all | 54 baseline image in the repository. Pixel tests are less robust than the |
| 55 alternatives listed above, because the rendering of a page is influenced by | 55 first two types, because the rendering of a page is influenced by |
| 56 many factors such as the host computer's graphics card and driver, the | 56 many factors such as the host computer's graphics card and driver, the |
| 57 platform's text rendering system, and various user-configurable operating | 57 platform's text rendering system, and various user-configurable operating |
| 58 system settings. For this reason, it is common for a pixel test to have a | 58 system settings. For this reason, it is common for a pixel test to have a |
| 59 different reference image for each platform that Blink is tested on. Pixel | 59 different reference image for each platform that Blink is tested on, and |
| 60 tests are least preferred, because the reference images are | 60 the reference images are |
| 61 [quite cumbersome to manage](./layout_test_expectations.md). | 61 [quite cumbersome to manage](./layout_test_expectations.md). You |
| 62 * *Dump Render Tree (DRT) Tests* output a textual representation of the render | 62 should only write a pixel test if you cannot use a reference test. By default |
| 63 a pixel test will also dump the layout tree as text output, so they are |
| 64 similar to ... |
| 65 * *Layout tree tests*, which output a textual representation of the layout |
| 63 tree, which is the key data structure in Blink's page rendering system. The | 66 tree, which is the key data structure in Blink's page rendering system. The |
| 64 test passes if the output matches a baseline text file in the repository. In | 67 test passes if the output matches a baseline text file in the repository. |
| 65 addition to their text result, DRT tests can also produce an image result | 68 Layout tree tests are used as a last resort to test the internal quirks of |
| 66 which is compared to an image baseline, similarly to pixel tests (described | 69 the implementation, and they should be avoided in favor of one of the earlier |
| 67 above). A DRT test with two results (text and image) passes if _both_ results | 70 options. |
| 68 match the baselines in the repository. DRT tests are less desirable than all | |
| 69 the alternatives, because they depend on a browser implementation detail. | |
| 70 | |
| 71 ## General Principles | 71 ## General Principles |
| 72 | 72 |
| 73 The principles below are adapted from | 73 The principles below are adapted from |
| 74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) | 74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc
s/test-format-guidelines.html) |
| 75 and | 75 and |
| 76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). | 76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri
ting%20Layout%20Tests%20for%20DumpRenderTree). |
| 77 | 77 |
| 78 *** note | 78 *** note |
| 79 This document intentionally uses _should_ a lot more than _must_, as defined in | 79 This document intentionally uses _should_ a lot more than _must_, as defined in |
| 80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a | 80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a |
| (...skipping 672 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 753 If that is not possible, make the test self-describing by including a textual | 753 If that is not possible, make the test self-describing by including a textual |
| 754 description of the desired (passing) outcome. | 754 description of the desired (passing) outcome. |
| 755 * Only use the red color or the word `FAIL` to highlight errors. This does not | 755 * Only use the red color or the word `FAIL` to highlight errors. This does not |
| 756 apply when testing the color red. | 756 apply when testing the color red. |
| 757 * 🚧 Use the | 757 * 🚧 Use the |
| 758 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the | 758 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the |
| 759 variance introduced by the platform's text rendering system. This does not | 759 variance introduced by the platform's text rendering system. This does not |
| 760 apply when testing text, text flow, font selection, font fallback, font | 760 apply when testing text, text flow, font selection, font fallback, font |
| 761 features, or other typographic information. | 761 features, or other typographic information. |
| 762 | 762 |
| 763 TODO: Document how to opt out of generating a layout tree when generating |
| 764 pixel results. |
| 765 |
| 763 *** promo | 766 *** promo |
| 764 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result | 767 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result |
| 765 will always be 800x600px, because test pages are rendered in an 800x600px | 768 will always be 800x600px, because test pages are rendered in an 800x600px |
| 766 viewport. Pixel tests that do not specifically cover scrolling should fit in an | 769 viewport. Pixel tests that do not specifically cover scrolling should fit in an |
| 767 800x600px viewport without creating scrollbars. | 770 800x600px viewport without creating scrollbars. |
| 768 *** | 771 *** |
| 769 | 772 |
| 770 *** promo | 773 *** promo |
| 771 The recommendation of using Ahem in pixel tests is being discussed on | 774 The recommendation of using Ahem in pixel tests is being discussed on |
| 772 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). | 775 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS
1E/discussion). |
| (...skipping 18 matching lines...) Expand all Loading... |
| 791 | 794 |
| 792 ### Tests that need to paint, raster, or draw a frame of intermediate output | 795 ### Tests that need to paint, raster, or draw a frame of intermediate output |
| 793 | 796 |
| 794 A layout test does not actually draw frames of output until the test exits. | 797 A layout test does not actually draw frames of output until the test exits. |
| 795 Tests that need to generate a painted frame can use | 798 Tests that need to generate a painted frame can use |
| 796 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a | 799 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a |
| 797 frame, then call the passed callback. There is also a library at | 800 frame, then call the passed callback. There is also a library at |
| 798 `fast/repaint/resources/text-based-repaint.js` to help with writing paint | 801 `fast/repaint/resources/text-based-repaint.js` to help with writing paint |
| 799 invalidation and repaint tests. | 802 invalidation and repaint tests. |
| 800 | 803 |
| 801 ## Dump Render Tree (DRT) Tests | 804 ## Layout tree tests |
| 802 | 805 |
| 803 A Dump Render Tree test renders a web page and produces up to two results, which | 806 A layout tree test renders a web page and produces up to two results, which |
| 804 are compared against baseline files: | 807 are compared against baseline files: |
| 805 | 808 |
| 806 * All tests output a textual representation of Blink's | 809 * All tests output a textual representation of Blink's |
| 807 [render tree](https://developers.google.com/web/fundamentals/performance/criti
cal-rendering-path/render-tree-construction), | 810 [layout tree](https://developers.google.com/web/fundamentals/performance/criti
cal-rendering-path/render-tree-construction) (called the render tree on that pag
e), |
| 808 which is compared against an `-expected.txt` text baseline. | 811 which is compared against an `-expected.txt` text baseline. |
| 809 * Some tests also output the image of the rendered page, which is compared | 812 * Some tests also output the image of the rendered page, which is compared |
| 810 against an `-expected.png` image baseline, using the same method as pixel | 813 against an `-expected.png` image baseline, using the same method as pixel |
| 811 tests. | 814 tests. |
| 812 | 815 |
| 813 TODO: Document the API used by DRT tests to opt out of producing image results. | 816 Whether you want a pixel test or a layout tree test depends on whether |
| 817 you care about the visual image, the details of how that image was |
| 818 constructed, or both. It is possible for multiple layout trees to produce |
| 819 the same pixel output, so it is important to make it clear in the test |
| 820 which outputs you really care about. |
| 814 | 821 |
| 815 A DRT test passes if _all_ of its results match their baselines. Like pixel | 822 TODO: Document the API used by layout tree tests to opt out of producing image |
| 816 tests, the output of DRT tests depends on platform-specific mechanisms, so DRT | 823 results. |
| 817 tests often require per-platform baselines. Furthermore, DRT tests depend on the | |
| 818 render tree data structure, which means that if we replace the render tree data | |
| 819 structure, we will need to look at each DRT test and consider whether it is | |
| 820 still meaningful. | |
| 821 | 824 |
| 822 For these reasons, DRT tests should **only** be used to cover aspects of the | 825 A layout tree test passes if _all_ of its results match their baselines. Like pi
xel |
| 823 layout code that can only be tested by looking at the render tree. Any | 826 tests, the output of layout tree tests depends on platform-specific details, |
| 824 combination of the other test types is preferable to a DRT test. DRT tests are | 827 so layout tree tests often require per-platform baselines. Furthermore, |
| 828 since the tests obviously depend on the layout tree structure, |
| 829 that means that if we change the layout tree you have to rebaseline each |
| 830 layout tree test to see if the results are still correct and whether the test |
| 831 is still meaningful. There are actually many cases where the layout tree |
| 832 output is misstated (i.e., wrong), because people didn't want to have to update |
| 833 existing baselines and tests. This is really unfortunate and confusing. |
| 834 |
| 835 For these reasons, layout tree tests should **only** be used to cover aspects |
| 836 of the layout code that can only be tested by looking at the layout tree. Any |
| 837 combination of the other test types is preferable to a layout tree test. |
| 838 Layout tree tests are |
| 825 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so | 839 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so |
| 826 the repository may have some unfortunate examples of DRT tests. | 840 the repository may have some unfortunate examples of layout tree tests. |
| 827 | 841 |
| 828 The following page is an example of a DRT test. | 842 |
| 843 The following page is an example of a layout tree test. |
| 829 | 844 |
| 830 ```html | 845 ```html |
| 831 <!doctype html> | 846 <!doctype html> |
| 832 <meta charset="utf-8"> | 847 <meta charset="utf-8"> |
| 833 <style> | 848 <style> |
| 834 body { font: 10px Ahem; } | 849 body { font: 10px Ahem; } |
| 835 span::after { | 850 span::after { |
| 836 content: "pass"; | 851 content: "pass"; |
| 837 color: green; | 852 color: green; |
| 838 } | 853 } |
| (...skipping 20 matching lines...) Expand all Loading... |
| 859 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] | 874 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] |
| 860 LayoutTextFragment (anonymous) at (430,0) size 40x10 | 875 LayoutTextFragment (anonymous) at (430,0) size 40x10 |
| 861 text run at (430,0) width 40: "pass" | 876 text run at (430,0) width 40: "pass" |
| 862 ``` | 877 ``` |
| 863 | 878 |
| 864 Notice that the test result above depends on the size of the `<p>` text. The | 879 Notice that the test result above depends on the size of the `<p>` text. The |
| 865 test page uses the Ahem font (introduced above), whose main design goal is | 880 test page uses the Ahem font (introduced above), whose main design goal is |
| 866 consistent cross-platform rendering. Had the test used another font, its text | 881 consistent cross-platform rendering. Had the test used another font, its text |
| 867 baseline would have depended on the fonts installed on the testing computer, and | 882 baseline would have depended on the fonts installed on the testing computer, and |
| 868 on the platform's font rendering system. Please follow the pixel tests | 883 on the platform's font rendering system. Please follow the pixel tests |
| 869 guidelines and write reliable DRT tests! | 884 guidelines and write reliable layout tree tests! |
| 870 | 885 |
| 871 WebKit's render tree is described in | 886 WebKit's layout tree is described in |
| 872 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) | 887 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) |
| 873 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. | 888 on WebKit's blog. Some of the concepts there still apply to Blink's layout tree. |
| 874 | 889 |
| 875 ## Directory Structure | 890 ## Directory Structure |
| 876 | 891 |
| 877 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently | 892 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently |
| 878 lacks a strict, formal structure. The following directories have special | 893 lacks a strict, formal structure. The following directories have special |
| 879 meaning: | 894 meaning: |
| 880 | 895 |
| 881 * The `http/` directory hosts tests that require an HTTP server (see above). | 896 * The `http/` directory hosts tests that require an HTTP server (see above). |
| 882 * The `resources/` subdirectory in every directory contains binary files, such | 897 * The `resources/` subdirectory in every directory contains binary files, such |
| 883 as media files, and code that is shared by multiple test files. | 898 as media files, and code that is shared by multiple test files. |
| 884 | 899 |
| 885 *** note | 900 *** note |
| 886 Some layout tests consist of a minimal HTML page that references a JavaScript | 901 Some layout tests consist of a minimal HTML page that references a JavaScript |
| 887 file in `resources/`. Please do not use this pattern for new tests, as it goes | 902 file in `resources/`. Please do not use this pattern for new tests, as it goes |
| 888 against the minimality principle. JavaScript and CSS files should only live in | 903 against the minimality principle. JavaScript and CSS files should only live in |
| 889 `resources/` if they are shared by at least two test files. | 904 `resources/` if they are shared by at least two test files. |
| 890 *** | 905 *** |
| OLD | NEW |