Chromium Code Reviews| OLD | NEW |
|---|---|
| 1 # Writing Layout Tests | 1 # Writing Layout Tests |
| 2 | 2 |
| 3 _Layout tests_ is a bit of a misnomer. This term is | 3 _Layout tests_ is a bit of a misnomer. This term is |
| 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), | 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), |
| 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, | 5 and we use it to refer to every test that is written as a Web page (HTML, SVG, |
| 6 or XHTML) and lives in | 6 or XHTML) and lives in |
| 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). | 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
| 8 | 8 |
| 9 [TOC] | 9 [TOC] |
| 10 | 10 |
| (...skipping 31 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 42 * *JavaScript Tests* are the layout test implementation of | 42 * *JavaScript Tests* are the layout test implementation of |
| 43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain | 43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
| 44 assertions written in JavaScript, and pass if the assertions evaluate to | 44 assertions written in JavaScript, and pass if the assertions evaluate to |
| 45 true. | 45 true. |
| 46 * *Reference Tests* render a test page and a reference page, and pass if the two | 46 * *Reference Tests* render a test page and a reference page, and pass if the two |
| 47 renderings are identical, according to a pixel-by-pixel comparison. These | 47 renderings are identical, according to a pixel-by-pixel comparison. These |
| 48 tests are less robust, harder to debug, and significantly slower than | 48 tests are less robust, harder to debug, and significantly slower than |
| 49 JavaScript tests, and are only used when JavaScript tests are insufficient, | 49 JavaScript tests, and are only used when JavaScript tests are insufficient, |
| 50 such as when testing paint code. | 50 such as when testing paint code. |
| 51 * *Pixel Tests* render a test page and compare the result against a pre-rendered | 51 * *Pixel Tests* render a test page and compare the result against a pre-rendered |
| 52 baseline image in the repository. Pixel tests are less robust than all | 52 baseline image in the repository. Pixel tests are less robust than the |
| 53 alternatives listed above, because the rendering of a page is influenced by | 53 first two types, because the rendering of a page is influenced by |
| 54 many factors such as the host computer's graphics card and driver, the | 54 many factors such as the host computer's graphics card and driver, the |
| 55 platform's text rendering system, and various user-configurable operating | 55 platform's text rendering system, and various user-configurable operating |
| 56 system settings. For this reason, it is common for a pixel test to have a | 56 system settings. For this reason, it is common for a pixel test to have a |
| 57 different reference image for each platform that Blink is tested on. Pixel | 57 different reference image for each platform that Blink is tested on, and |
| 58 tests are least preferred, because the reference images are | 58 the reference images are |
| 59 [quite cumbersome to manage](./layout_test_expectations.md). | 59 [quite cumbersome to manage](./layout_test_expectations.md). You |
| 60 * *Dump Render Tree (DRT) Tests* output a textual representation of the render | 60 should only write a pixel test if you cannot use a reference test. By default |
| 61 a pixel test will also dump the render tree as text output, so they are | |
| 62 similar to ... | |
| 63 * *Render tree tests*, which output a textual representation of the render | |
|
cbiesinger
2016/12/02 22:15:22
We call it Layout Tree now, fwiw
| |
| 61 tree, which is the key data structure in Blink's page rendering system. The | 64 tree, which is the key data structure in Blink's page rendering system. The |
| 62 test passes if the output matches a baseline text file in the repository. In | 65 test passes if the output matches a baseline text file in the repository. |
| 63 addition to their text result, DRT tests can also produce an image result | 66 Render tree tests are used as a last resort to test the internal quirks of |
| 64 which is compared to an image baseline, similarly to pixel tests (described | 67 the implementation, and they should be avoided in favor of one of the earlier |
| 65 above). A DRT test with two results (text and image) passes if _both_ results | 68 options. |
| 66 match the baselines in the repository. DRT tests are less desirable than all | |
| 67 the alternatives, because they depend on a browser implementation detail. | |
| 68 | |
| 69 ## General Principles | 69 ## General Principles |
| 70 | 70 |
| 71 The principles below are adapted from | 71 The principles below are adapted from |
| 72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) | 72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) |
| 73 and | 73 and |
| 74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). | 74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). |
| 75 | 75 |
| 76 * Tests should be **concise**, without compromising on the principles below. | 76 * Tests should be **concise**, without compromising on the principles below. |
| 77 Every element and piece of code on the page should be necessary and relevant | 77 Every element and piece of code on the page should be necessary and relevant |
| 78 to what is being tested. For example, don't build a fully functional signup | 78 to what is being tested. For example, don't build a fully functional signup |
| (...skipping 609 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 688 | 688 |
| 689 ### Tests that need to paint, raster, or draw a frame of intermediate output | 689 ### Tests that need to paint, raster, or draw a frame of intermediate output |
| 690 | 690 |
| 691 A layout test does not actually draw frames of output until the test exits. | 691 A layout test does not actually draw frames of output until the test exits. |
| 692 Tests that need to generate a painted frame can use | 692 Tests that need to generate a painted frame can use |
| 693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a | 693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a |
| 694 frame, then call the passed callback. There is also a library at | 694 frame, then call the passed callback. There is also a library at |
| 695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint | 695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint |
| 696 invalidation and repaint tests. | 696 invalidation and repaint tests. |
| 697 | 697 |
| 698 ## Dump Render Tree (DRT) Tests | 698 ## Render tree Tests |
| 699 | 699 |
| 700 A Dump Render Tree test renders a web page and produces up to two results, which | 700 A render tree test renders a web page and produces up to two results, which |
| 701 are compared against baseline files: | 701 are compared against baseline files: |
| 702 | 702 |
| 703 * All tests output a textual representation of Blink's | 703 * All tests output a textual representation of Blink's |
| 704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), | 704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), |
| 705 which is compared against an `-expected.txt` text baseline. | 705 which is compared against an `-expected.txt` text baseline. |
| 706 * Some tests also output the image of the rendered page, which is compared | 706 * Some tests also output the image of the rendered page, which is compared |
| 707 against an `-expected.png` image baseline, using the same method as pixel | 707 against an `-expected.png` image baseline, using the same method as pixel |
| 708 tests. | 708 tests. |
| 709 | 709 |
| 710 TODO: Document the API used by DRT tests to opt out of producing image results. | 710 Whether you want a pixel test or a render tree test depends on whether |
| 711 you care about the visual image, the details of how that image was | |
| 712 constructed, or both. It is possible for multiple render trees to produce | |
| 713 the same pixel output, so it is important to make it clear in the test | |
| 714 which outputs you really care about. | |
| 711 | 715 |
| 712 A DRT test passes if _all_ of its results match their baselines. Like pixel | 716 TODO: Document the API used by render tree tests to opt out of producing image |
| 713 tests, the output of DRT tests depends on platform-specific mechanisms, so DRT | 717 results. |
|
m_gsnedders
2016/12/02 14:16:44
Given the intro above says that by default "a pixe
| |
| 714 tests often require per-platform baselines. Furthermore, DRT tests depend on the | |
| 715 render tree data structure, which means that if we replace the render tree data | |
| 716 structure, we will need to look at each DRT test and consider whether it is | |
| 717 still meaningful. | |
| 718 | 718 |
| 719 For these reasons, DRT tests should **only** be used to cover aspects of the | 719 A render tree test passes if _all_ of its results match their baselines. Like pi xel |
| 720 layout code that can only be tested by looking at the render tree. Any | 720 tests, the output of render tree tests depends on platform-specific details, |
| 721 combination of the other test types is preferable to a DRT test. DRT tests are | 721 so render tree tests often require per-platform baselines. Furthermore, |
| 722 since the tests obviously depend on the render tree structure, | |
| 723 that means that if we change the render tree you have to rebaseline each | |
| 724 render tree test to see if the results are still correct and whether the test | |
| 725 is still meaningful. There are actually many cases where the render tree | |
|
pwnall
2016/12/02 03:20:48
Could "many cases" link to a crbug that tracks rem
| |
| 726 output is misstated (i.e., wrong), because people didn't want to have to update | |
| 727 existing baselines and tests. This is really unfortunate and confusing. | |
| 728 | |
| 729 For these reasons, render tree tests should **only** be used to cover aspects | |
| 730 of the layout code that can only be tested by looking at the render tree. Any | |
| 731 combination of the other test types is preferable to a render tree test. | |
| 732 render tree tests are | |
|
m_gsnedders
2016/12/02 14:16:44
(grammar): needs a capital R
| |
| 722 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so | 733 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so |
| 723 the repository may have some unfortunate examples of DRT tests. | 734 the repository may have some unfortunate examples of render tree tests. |
| 724 | 735 |
| 725 The following page is an example of a DRT test. | 736 |
| 737 The following page is an example of a render tree test. | |
| 726 | 738 |
| 727 ```html | 739 ```html |
| 728 <!doctype html> | 740 <!doctype html> |
| 729 <meta charset="utf-8"> | 741 <meta charset="utf-8"> |
| 730 <style> | 742 <style> |
| 731 body { font: 10px Ahem; } | 743 body { font: 10px Ahem; } |
| 732 span::after { | 744 span::after { |
| 733 content: "pass"; | 745 content: "pass"; |
| 734 color: green; | 746 color: green; |
| 735 } | 747 } |
| (...skipping 20 matching lines...) Expand all Loading... | |
| 756 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] | 768 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] |
| 757 LayoutTextFragment (anonymous) at (430,0) size 40x10 | 769 LayoutTextFragment (anonymous) at (430,0) size 40x10 |
| 758 text run at (430,0) width 40: "pass" | 770 text run at (430,0) width 40: "pass" |
| 759 ``` | 771 ``` |
| 760 | 772 |
| 761 Notice that the test result above depends on the size of the `<p>` text. The | 773 Notice that the test result above depends on the size of the `<p>` text. The |
| 762 test page uses the Ahem font (introduced above), whose main design goal is | 774 test page uses the Ahem font (introduced above), whose main design goal is |
| 763 consistent cross-platform rendering. Had the test used another font, its text | 775 consistent cross-platform rendering. Had the test used another font, its text |
| 764 baseline would have depended on the fonts installed on the testing computer, and | 776 baseline would have depended on the fonts installed on the testing computer, and |
| 765 on the platform's font rendering system. Please follow the pixel tests | 777 on the platform's font rendering system. Please follow the pixel tests |
| 766 guidelines and write reliable DRT tests! | 778 guidelines and write reliable render tree tests! |
| 767 | 779 |
| 768 WebKit's render tree is described in | 780 WebKit's render tree is described in |
| 769 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) | 781 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) |
| 770 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. | 782 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. |
| 771 | 783 |
| 772 ## Directory Structure | 784 ## Directory Structure |
| 773 | 785 |
| 774 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently | 786 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently |
| 775 lacks a strict, formal structure. The following directories have special | 787 lacks a strict, formal structure. The following directories have special |
| 776 meaning: | 788 meaning: |
| 777 | 789 |
| 778 * The `http/` directory hosts tests that require an HTTP server (see above). | 790 * The `http/` directory hosts tests that require an HTTP server (see above). |
| 779 * The `resources/` subdirectory in every directory contains binary files, such | 791 * The `resources/` subdirectory in every directory contains binary files, such |
| 780 as media files, and code that is shared by multiple test files. | 792 as media files, and code that is shared by multiple test files. |
| 781 | 793 |
| 782 *** note | 794 *** note |
| 783 Some layout tests consist of a minimal HTML page that references a JavaScript | 795 Some layout tests consist of a minimal HTML page that references a JavaScript |
| 784 file in `resources/`. Please do not use this pattern for new tests, as it goes | 796 file in `resources/`. Please do not use this pattern for new tests, as it goes |
| 785 against the minimality principle. JavaScript and CSS files should only live in | 797 against the minimality principle. JavaScript and CSS files should only live in |
| 786 `resources/` if they are shared by at least two test files. | 798 `resources/` if they are shared by at least two test files. |
| 787 *** | 799 *** |
| OLD | NEW |