Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(244)

Side by Side Diff: docs/testing/writing_layout_tests.md

Issue 2547463003: Tweak the descriptions of the various types of layout tests. (Closed)
Patch Set: rework, remove other DRT references Created 4 years ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Writing Layout Tests 1 # Writing Layout Tests
2 2
3 _Layout tests_ is a bit of a misnomer. This term is 3 _Layout tests_ is a bit of a misnomer. This term is
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /),
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, 5 and we use it to refer to every test that is written as a Web page (HTML, SVG,
6 or XHTML) and lives in 6 or XHTML) and lives in
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8 8
9 [TOC] 9 [TOC]
10 10
(...skipping 31 matching lines...) Expand 10 before | Expand all | Expand 10 after
42 * *JavaScript Tests* are the layout test implementation of 42 * *JavaScript Tests* are the layout test implementation of
43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain 43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
44 assertions written in JavaScript, and pass if the assertions evaluate to 44 assertions written in JavaScript, and pass if the assertions evaluate to
45 true. 45 true.
46 * *Reference Tests* render a test page and a reference page, and pass if the two 46 * *Reference Tests* render a test page and a reference page, and pass if the two
47 renderings are identical, according to a pixel-by-pixel comparison. These 47 renderings are identical, according to a pixel-by-pixel comparison. These
48 tests are less robust, harder to debug, and significantly slower than 48 tests are less robust, harder to debug, and significantly slower than
49 JavaScript tests, and are only used when JavaScript tests are insufficient, 49 JavaScript tests, and are only used when JavaScript tests are insufficient,
50 such as when testing paint code. 50 such as when testing paint code.
51 * *Pixel Tests* render a test page and compare the result against a pre-rendered 51 * *Pixel Tests* render a test page and compare the result against a pre-rendered
52 baseline image in the repository. Pixel tests are less robust than all 52 baseline image in the repository. Pixel tests are less robust than the
53 alternatives listed above, because the rendering of a page is influenced by 53 first two types, because the rendering of a page is influenced by
54 many factors such as the host computer's graphics card and driver, the 54 many factors such as the host computer's graphics card and driver, the
55 platform's text rendering system, and various user-configurable operating 55 platform's text rendering system, and various user-configurable operating
56 system settings. For this reason, it is common for a pixel test to have a 56 system settings. For this reason, it is common for a pixel test to have a
57 different reference image for each platform that Blink is tested on. Pixel 57 different reference image for each platform that Blink is tested on, and
58 tests are least preferred, because the reference images are 58 the reference images are
59 [quite cumbersome to manage](./layout_test_expectations.md). 59 [quite cumbersome to manage](./layout_test_expectations.md). You
60 * *Dump Render Tree (DRT) Tests* output a textual representation of the render 60 should only write a pixel test if you cannot use a reference test. By default
61 a pixel test will also dump the render tree as text output, so they are
62 similar to ...
63 * *Render tree tests*, which output a textual representation of the render
cbiesinger 2016/12/02 22:15:22 We call it Layout Tree now, fwiw
61 tree, which is the key data structure in Blink's page rendering system. The 64 tree, which is the key data structure in Blink's page rendering system. The
62 test passes if the output matches a baseline text file in the repository. In 65 test passes if the output matches a baseline text file in the repository.
63 addition to their text result, DRT tests can also produce an image result 66 Render tree tests are used as a last resort to test the internal quirks of
64 which is compared to an image baseline, similarly to pixel tests (described 67 the implementation, and they should be avoided in favor of one of the earlier
65 above). A DRT test with two results (text and image) passes if _both_ results 68 options.
66 match the baselines in the repository. DRT tests are less desirable than all
67 the alternatives, because they depend on a browser implementation detail.
68
69 ## General Principles 69 ## General Principles
70 70
71 The principles below are adapted from 71 The principles below are adapted from
72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) 72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html)
73 and 73 and
74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). 74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree).
75 75
76 * Tests should be **concise**, without compromising on the principles below. 76 * Tests should be **concise**, without compromising on the principles below.
77 Every element and piece of code on the page should be necessary and relevant 77 Every element and piece of code on the page should be necessary and relevant
78 to what is being tested. For example, don't build a fully functional signup 78 to what is being tested. For example, don't build a fully functional signup
(...skipping 609 matching lines...) Expand 10 before | Expand all | Expand 10 after
688 688
689 ### Tests that need to paint, raster, or draw a frame of intermediate output 689 ### Tests that need to paint, raster, or draw a frame of intermediate output
690 690
691 A layout test does not actually draw frames of output until the test exits. 691 A layout test does not actually draw frames of output until the test exits.
692 Tests that need to generate a painted frame can use 692 Tests that need to generate a painted frame can use
693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a 693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a
694 frame, then call the passed callback. There is also a library at 694 frame, then call the passed callback. There is also a library at
695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint 695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint
696 invalidation and repaint tests. 696 invalidation and repaint tests.
697 697
698 ## Dump Render Tree (DRT) Tests 698 ## Render tree Tests
699 699
700 A Dump Render Tree test renders a web page and produces up to two results, which 700 A render tree test renders a web page and produces up to two results, which
701 are compared against baseline files: 701 are compared against baseline files:
702 702
703 * All tests output a textual representation of Blink's 703 * All tests output a textual representation of Blink's
704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), 704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction),
705 which is compared against an `-expected.txt` text baseline. 705 which is compared against an `-expected.txt` text baseline.
706 * Some tests also output the image of the rendered page, which is compared 706 * Some tests also output the image of the rendered page, which is compared
707 against an `-expected.png` image baseline, using the same method as pixel 707 against an `-expected.png` image baseline, using the same method as pixel
708 tests. 708 tests.
709 709
710 TODO: Document the API used by DRT tests to opt out of producing image results. 710 Whether you want a pixel test or a render tree test depends on whether
711 you care about the visual image, the details of how that image was
712 constructed, or both. It is possible for multiple render trees to produce
713 the same pixel output, so it is important to make it clear in the test
714 which outputs you really care about.
711 715
712 A DRT test passes if _all_ of its results match their baselines. Like pixel 716 TODO: Document the API used by render tree tests to opt out of producing image
713 tests, the output of DRT tests depends on platform-specific mechanisms, so DRT 717 results.
m_gsnedders 2016/12/02 14:16:44 Given the intro above says that by default "a pixe
714 tests often require per-platform baselines. Furthermore, DRT tests depend on the
715 render tree data structure, which means that if we replace the render tree data
716 structure, we will need to look at each DRT test and consider whether it is
717 still meaningful.
718 718
719 For these reasons, DRT tests should **only** be used to cover aspects of the 719 A render tree test passes if _all_ of its results match their baselines. Like pi xel
720 layout code that can only be tested by looking at the render tree. Any 720 tests, the output of render tree tests depends on platform-specific details,
721 combination of the other test types is preferable to a DRT test. DRT tests are 721 so render tree tests often require per-platform baselines. Furthermore,
722 since the tests obviously depend on the render tree structure,
723 that means that if we change the render tree you have to rebaseline each
724 render tree test to see if the results are still correct and whether the test
725 is still meaningful. There are actually many cases where the render tree
pwnall 2016/12/02 03:20:48 Could "many cases" link to a crbug that tracks rem
726 output is misstated (i.e., wrong), because people didn't want to have to update
727 existing baselines and tests. This is really unfortunate and confusing.
728
729 For these reasons, render tree tests should **only** be used to cover aspects
730 of the layout code that can only be tested by looking at the render tree. Any
731 combination of the other test types is preferable to a render tree test.
732 render tree tests are
m_gsnedders 2016/12/02 14:16:44 (grammar): needs a capital R
722 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so 733 [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so
723 the repository may have some unfortunate examples of DRT tests. 734 the repository may have some unfortunate examples of render tree tests.
724 735
725 The following page is an example of a DRT test. 736
737 The following page is an example of a render tree test.
726 738
727 ```html 739 ```html
728 <!doctype html> 740 <!doctype html>
729 <meta charset="utf-8"> 741 <meta charset="utf-8">
730 <style> 742 <style>
731 body { font: 10px Ahem; } 743 body { font: 10px Ahem; }
732 span::after { 744 span::after {
733 content: "pass"; 745 content: "pass";
734 color: green; 746 color: green;
735 } 747 }
(...skipping 20 matching lines...) Expand all
756 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] 768 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000]
757 LayoutTextFragment (anonymous) at (430,0) size 40x10 769 LayoutTextFragment (anonymous) at (430,0) size 40x10
758 text run at (430,0) width 40: "pass" 770 text run at (430,0) width 40: "pass"
759 ``` 771 ```
760 772
761 Notice that the test result above depends on the size of the `<p>` text. The 773 Notice that the test result above depends on the size of the `<p>` text. The
762 test page uses the Ahem font (introduced above), whose main design goal is 774 test page uses the Ahem font (introduced above), whose main design goal is
763 consistent cross-platform rendering. Had the test used another font, its text 775 consistent cross-platform rendering. Had the test used another font, its text
764 baseline would have depended on the fonts installed on the testing computer, and 776 baseline would have depended on the fonts installed on the testing computer, and
765 on the platform's font rendering system. Please follow the pixel tests 777 on the platform's font rendering system. Please follow the pixel tests
766 guidelines and write reliable DRT tests! 778 guidelines and write reliable render tree tests!
767 779
768 WebKit's render tree is described in 780 WebKit's render tree is described in
769 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) 781 [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/)
770 on WebKit's blog. Some of the concepts there still apply to Blink's render tree. 782 on WebKit's blog. Some of the concepts there still apply to Blink's render tree.
771 783
772 ## Directory Structure 784 ## Directory Structure
773 785
774 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently 786 The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
775 lacks a strict, formal structure. The following directories have special 787 lacks a strict, formal structure. The following directories have special
776 meaning: 788 meaning:
777 789
778 * The `http/` directory hosts tests that require an HTTP server (see above). 790 * The `http/` directory hosts tests that require an HTTP server (see above).
779 * The `resources/` subdirectory in every directory contains binary files, such 791 * The `resources/` subdirectory in every directory contains binary files, such
780 as media files, and code that is shared by multiple test files. 792 as media files, and code that is shared by multiple test files.
781 793
782 *** note 794 *** note
783 Some layout tests consist of a minimal HTML page that references a JavaScript 795 Some layout tests consist of a minimal HTML page that references a JavaScript
784 file in `resources/`. Please do not use this pattern for new tests, as it goes 796 file in `resources/`. Please do not use this pattern for new tests, as it goes
785 against the minimality principle. JavaScript and CSS files should only live in 797 against the minimality principle. JavaScript and CSS files should only live in
786 `resources/` if they are shared by at least two test files. 798 `resources/` if they are shared by at least two test files.
787 *** 799 ***
OLDNEW
« no previous file with comments | « no previous file | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698