Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(785)

Side by Side Diff: docs/testing/writing_layout_tests.md

Issue 2547463003: Tweak the descriptions of the various types of layout tests. (Closed)
Patch Set: s/render tree/layout tree Created 3 years, 11 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Writing Layout Tests 1 # Writing Layout Tests
2 2
3 _Layout tests_ is a bit of a misnomer. This term is 3 _Layout tests_ is a bit of a misnomer. This term is
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /),
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, 5 and we use it to refer to every test that is written as a Web page (HTML, SVG,
6 or XHTML) and lives in 6 or XHTML) and lives in
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8 8
9 [TOC] 9 [TOC]
10 10
(...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after
44 * *JavaScript Tests* are the layout test implementation of 44 * *JavaScript Tests* are the layout test implementation of
45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain 45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
46 assertions written in JavaScript, and pass if the assertions evaluate to 46 assertions written in JavaScript, and pass if the assertions evaluate to
47 true. 47 true.
48 * *Reference Tests* render a test page and a reference page, and pass if the two 48 * *Reference Tests* render a test page and a reference page, and pass if the two
49 renderings are identical, according to a pixel-by-pixel comparison. These 49 renderings are identical, according to a pixel-by-pixel comparison. These
50 tests are less robust, harder to debug, and significantly slower than 50 tests are less robust, harder to debug, and significantly slower than
51 JavaScript tests, and are only used when JavaScript tests are insufficient, 51 JavaScript tests, and are only used when JavaScript tests are insufficient,
52 such as when testing paint code. 52 such as when testing paint code.
53 * *Pixel Tests* render a test page and compare the result against a pre-rendered 53 * *Pixel Tests* render a test page and compare the result against a pre-rendered
54 baseline image in the repository. Pixel tests are less robust than all 54 baseline image in the repository. Pixel tests are less robust than the
55 alternatives listed above, because the rendering of a page is influenced by 55 first two types, because the rendering of a page is influenced by
56 many factors such as the host computer's graphics card and driver, the 56 many factors such as the host computer's graphics card and driver, the
57 platform's text rendering system, and various user-configurable operating 57 platform's text rendering system, and various user-configurable operating
58 system settings. For this reason, it is common for a pixel test to have a 58 system settings. For this reason, it is common for a pixel test to have a
59 different reference image for each platform that Blink is tested on. Pixel 59 different reference image for each platform that Blink is tested on, and
60 tests are least preferred, because the reference images are 60 the reference images are
61 [quite cumbersome to manage](./layout_test_expectations.md). 61 [quite cumbersome to manage](./layout_test_expectations.md). You
62 * *Dump Render Tree (DRT) Tests* output a textual representation of the render 62 should only write a pixel test if you cannot use a reference test. By default
63 a pixel test will also dump the layout tree as text output, so they are
64 similar to ...
65 * *Layout tree tests*, which output a textual representation of the layout
63 tree, which is the key data structure in Blink's page rendering system. The 66 tree, which is the key data structure in Blink's page rendering system. The
64 test passes if the output matches a baseline text file in the repository. In 67 test passes if the output matches a baseline text file in the repository.
65 addition to their text result, DRT tests can also produce an image result 68 Layout tree tests are used as a last resort to test the internal quirks of
66 which is compared to an image baseline, similarly to pixel tests (described 69 the implementation, and they should be avoided in favor of one of the earlier
67 above). A DRT test with two results (text and image) passes if _both_ results 70 options.
68 match the baselines in the repository. DRT tests are less desirable than all
69 the alternatives, because they depend on a browser implementation detail.
70
71 ## General Principles 71 ## General Principles
72 72
73 The principles below are adapted from 73 The principles below are adapted from
74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) 74 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html)
75 and 75 and
76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). 76 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree).
77 77
78 *** note 78 *** note
79 This document intentionally uses _should_ a lot more than _must_, as defined in 79 This document intentionally uses _should_ a lot more than _must_, as defined in
80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a 80 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a
(...skipping 672 matching lines...) Expand 10 before | Expand all | Expand 10 after
753 If that is not possible, make the test self-describing by including a textual 753 If that is not possible, make the test self-describing by including a textual
754 description of the desired (passing) outcome. 754 description of the desired (passing) outcome.
755 * Only use the red color or the word `FAIL` to highlight errors. This does not 755 * Only use the red color or the word `FAIL` to highlight errors. This does not
756 apply when testing the color red. 756 apply when testing the color red.
757 * 🚧 Use the 757 * 🚧 Use the
758 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the 758 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the
759 variance introduced by the platform's text rendering system. This does not 759 variance introduced by the platform's text rendering system. This does not
760 apply when testing text, text flow, font selection, font fallback, font 760 apply when testing text, text flow, font selection, font fallback, font
761 features, or other typographic information. 761 features, or other typographic information.
762 762
763 TODO: Document how to opt out of generating a layout tree when generating
764 pixel results.
765
763 *** promo 766 *** promo
764 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result 767 When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result
765 will always be 800x600px, because test pages are rendered in an 800x600px 768 will always be 800x600px, because test pages are rendered in an 800x600px
766 viewport. Pixel tests that do not specifically cover scrolling should fit in an 769 viewport. Pixel tests that do not specifically cover scrolling should fit in an
767 800x600px viewport without creating scrollbars. 770 800x600px viewport without creating scrollbars.
768 *** 771 ***
769 772
770 *** promo 773 *** promo
771 The recommendation of using Ahem in pixel tests is being discussed on 774 The recommendation of using Ahem in pixel tests is being discussed on
772 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion). 775 [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS 1E/discussion).
(...skipping 18 matching lines...) Expand all
791 794
792 ### Tests that need to paint, raster, or draw a frame of intermediate output 795 ### Tests that need to paint, raster, or draw a frame of intermediate output
793 796
794 A layout test does not actually draw frames of output until the test exits. 797 A layout test does not actually draw frames of output until the test exits.
795 Tests that need to generate a painted frame can use 798 Tests that need to generate a painted frame can use
796 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a 799 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a
797 frame, then call the passed callback. There is also a library at 800 frame, then call the passed callback. There is also a library at
798 `fast/repaint/resources/text-based-repaint.js` to help with writing paint 801 `fast/repaint/resources/text-based-repaint.js` to help with writing paint
799 invalidation and repaint tests. 802 invalidation and repaint tests.
800 803
801 ## Dump Render Tree (DRT) Tests 804 ## Layout tree tests
802 805
803 A Dump Render Tree test renders a web page and produces up to two results, which 806 A layout tree test renders a web page and produces up to two results, which
804 are compared against baseline files: 807 are compared against baseline files:
805 808
806 * All tests output a textual representation of Blink's 809 * All tests output a textual representation of Blink's
807 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), 810 [layout tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction) (called the render tree on that pag e),
808 which is compared against an `-expected.txt` text baseline. 811 which is compared against an `-expected.txt` text baseline.
809 * Some tests also output the image of the rendered page, which is compared 812 * Some tests also output the image of the rendered page, which is compared
810 against an `-expected.png` image baseline, using the same method as pixel 813 against an `-expected.png` image baseline, using the same method as pixel
811 tests. 814 tests.
812 815
813 TODO: Document the API used by DRT tests to opt out of producing image results. 816 Whether you want a pixel test or a layout tree test depends on whether
817 you care about the visual image, the details of how that image was
818 constructed, or both. It is possible for multiple layout trees to produce
819 the same pixel output, so it is important to make it clear in the test
820 which outputs you really care about.
814 821
815 A DRT test passes if _all_ of its results match their baselines. Like pixel 822 TODO: Document the API used by layout tree tests to opt out of producing image
816 tests, the output of DRT tests depends on platform-specific mechanisms, so DRT 823 results.
817 tests often require per-platform baselines. Furthermore, DRT tests depend on the
818 render tree data structure, which means that if we replace the render tree data
819 structure, we will need to look at each DRT test and consider whether it is
820 still meaningful.
821 824
822 For these reasons, DRT tests should **only** be used to cover aspects of the 825 A layout tree test passes if _all_ of its results match their baselines. Like pi xel
823 layout code that can only be tested by looking at the render tree. Any 826 tests, the output of layout tree tests depends on platform-specific details,
824 combination of the other test types is preferable to a DRT test. DRT tests are 827 so layout tree tests often require per-platform baselines. Furthermore,
828 since the tests obviously depend on the layout tree structure,
829 that means that if we change the layout tree you have to rebaseline each
830 layout tree test to see if the results are still correct and whether the test
831 is still meaningful. There are actually many cases where the layout tree
832 output is misstated (i.e., wrong), because people didn't want to have to update
833 existing baselines and tests. This is really unfortunate and confusing.
834
835 For these reasons, layout tree tests should **only** be used to cover aspects
836 of the layout code that can only be tested by looking at the layout tree. Any
837 combination of the other test types is preferable to a layout tree test.
838 Layout tree tests are
825 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so 839 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so
826 the repository may have some unfortunate examples of DRT tests. 840 the repository may have some unfortunate examples of layout tree tests.
827 841
828 The following page is an example of a DRT test. 842
843 The following page is an example of a layout tree test.
829 844
830 ```html 845 ```html
831 <!doctype html> 846 <!doctype html>
832 <meta charset="utf-8"> 847 <meta charset="utf-8">
833 <style> 848 <style>
834 body { font: 10px Ahem; } 849 body { font: 10px Ahem; }
835 span::after { 850 span::after {
836 content: "pass"; 851 content: "pass";
837 color: green; 852 color: green;
838 } 853 }
(...skipping 20 matching lines...) Expand all
859 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] 874 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000]
860 LayoutTextFragment (anonymous) at (430,0) size 40x10 875 LayoutTextFragment (anonymous) at (430,0) size 40x10
861 text run at (430,0) width 40: "pass" 876 text run at (430,0) width 40: "pass"
862 ``` 877 ```
863 878
864 Notice that the test result above depends on the size of the `<p>` text. The 879 Notice that the test result above depends on the size of the `<p>` text. The
865 test page uses the Ahem font (introduced above), whose main design goal is 880 test page uses the Ahem font (introduced above), whose main design goal is
866 consistent cross-platform rendering. Had the test used another font, its text 881 consistent cross-platform rendering. Had the test used another font, its text
867 baseline would have depended on the fonts installed on the testing computer, and 882 baseline would have depended on the fonts installed on the testing computer, and
868 on the platform's font rendering system. Please follow the pixel tests 883 on the platform's font rendering system. Please follow the pixel tests
869 guidelines and write reliable DRT tests! 884 guidelines and write reliable layout tree tests!
870 885
871 WebKit's render tree is described in 886 WebKit's layout tree is described in
872 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) 887 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/)
873 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. 888 on WebKit's blog. Some of the concepts there still apply to Blink's layout tree.
874 889
875 ## Directory Structure 890 ## Directory Structure
876 891
877 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently 892 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
878 lacks a strict, formal structure. The following directories have special 893 lacks a strict, formal structure. The following directories have special
879 meaning: 894 meaning:
880 895
881 * The `http/` directory hosts tests that require an HTTP server (see above). 896 * The `http/` directory hosts tests that require an HTTP server (see above).
882 * The `resources/` subdirectory in every directory contains binary files, such 897 * The `resources/` subdirectory in every directory contains binary files, such
883 as media files, and code that is shared by multiple test files. 898 as media files, and code that is shared by multiple test files.
884 899
885 *** note 900 *** note
886 Some layout tests consist of a minimal HTML page that references a JavaScript 901 Some layout tests consist of a minimal HTML page that references a JavaScript
887 file in `resources/`. Please do not use this pattern for new tests, as it goes 902 file in `resources/`. Please do not use this pattern for new tests, as it goes
888 against the minimality principle. JavaScript and CSS files should only live in 903 against the minimality principle. JavaScript and CSS files should only live in
889 `resources/` if they are shared by at least two test files. 904 `resources/` if they are shared by at least two test files.
890 *** 905 ***
OLDNEW
« no previous file with comments | « no previous file | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698