Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(148)

Side by Side Diff: docs/testing/writing_layout_tests.md

Issue 2547463003: Tweak the descriptions of the various types of layout tests. (Closed)
Patch Set: Created 4 years ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Writing Layout Tests 1 # Writing Layout Tests
2 2
3 _Layout tests_ is a bit of a misnomer. This term is 3 _Layout tests_ is a bit of a misnomer. This term is
4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /), 4 [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory /),
5 and we use it to refer to every test that is written as a Web page (HTML, SVG, 5 and we use it to refer to every test that is written as a Web page (HTML, SVG,
6 or XHTML) and lives in 6 or XHTML) and lives in
7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). 7 [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8 8
9 [TOC] 9 [TOC]
10 10
(...skipping 31 matching lines...) Expand 10 before | Expand all | Expand 10 after
42 * *JavaScript Tests* are the layout test implementation of 42 * *JavaScript Tests* are the layout test implementation of
43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain 43 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
44 assertions written in JavaScript, and pass if the assertions evaluate to 44 assertions written in JavaScript, and pass if the assertions evaluate to
45 true. 45 true.
46 * *Reference Tests* render a test page and a reference page, and pass if the two 46 * *Reference Tests* render a test page and a reference page, and pass if the two
47 renderings are identical, according to a pixel-by-pixel comparison. These 47 renderings are identical, according to a pixel-by-pixel comparison. These
48 tests are less robust, harder to debug, and significantly slower than 48 tests are less robust, harder to debug, and significantly slower than
49 JavaScript tests, and are only used when JavaScript tests are insufficient, 49 JavaScript tests, and are only used when JavaScript tests are insufficient,
50 such as when testing paint code. 50 such as when testing paint code.
51 * *Pixel Tests* render a test page and compare the result against a pre-rendered 51 * *Pixel Tests* render a test page and compare the result against a pre-rendered
52 baseline image in the repository. Pixel tests are less robust than all 52 baseline image in the repository. Pixel tests are less robust than the
53 alternatives listed above, because the rendering of a page is influenced by 53 first two types, because the rendering of a page is influenced by
pwnall 2016/12/02 02:35:10 Thanks for catching this inconsistency!
54 many factors such as the host computer's graphics card and driver, the 54 many factors such as the host computer's graphics card and driver, the
55 platform's text rendering system, and various user-configurable operating 55 platform's text rendering system, and various user-configurable operating
56 system settings. For this reason, it is common for a pixel test to have a 56 system settings. For this reason, it is common for a pixel test to have a
57 different reference image for each platform that Blink is tested on. Pixel 57 different reference image for each platform that Blink is tested on, and
58 tests are least preferred, because the reference images are 58 the reference images are
59 [quite cumbersome to manage](./layout_test_expectations.md). 59 [quite cumbersome to manage](./layout_test_expectations.md). You
60 * *Dump Render Tree (DRT) Tests* output a textual representation of the render 60 should only write a pixel test if you cannot use a reference test. By default
61 a pixel test will also dump the render tree as text output, so they are
62 similar to ...
63 * *Render tree tests*, which output a textual representation of the render
61 tree, which is the key data structure in Blink's page rendering system. The 64 tree, which is the key data structure in Blink's page rendering system. The
62 test passes if the output matches a baseline text file in the repository. In 65 test passes if the output matches a baseline text file in the repository.
63 addition to their text result, DRT tests can also produce an image result 66 Render tree tests are used as a last resort to test the internal quirks of
64 which is compared to an image baseline, similarly to pixel tests (described 67 the implementation, and they should be avoided in favor of one of the earlier
65 above). A DRT test with two results (text and image) passes if _both_ results 68 options. Most render tree tests also produce image results, and so these
pwnall 2016/12/02 02:35:10 I think that the text starting at "Most render tre
Dirk Pranke 2016/12/02 02:44:54 Okay.
66 match the baselines in the repository. DRT tests are less desirable than all 69 tests are easily confused with Pixel tests. The distinction lies in whether
67 the alternatives, because they depend on a browser implementation detail. 70 you care about the visual image, the details of how that image was
71 constructed, or both. It is possible for multiple render trees to produce
72 the same pixel output, so it is important to make it clear in the test
73 which outputs you really care about.
68 74
69 ## General Principles 75 ## General Principles
70 76
71 The principles below are adapted from 77 The principles below are adapted from
72 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html) 78 [Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/doc s/test-format-guidelines.html)
73 and 79 and
74 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree). 80 [WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Wri ting%20Layout%20Tests%20for%20DumpRenderTree).
75 81
76 * Tests should be **concise**, without compromising on the principles below. 82 * Tests should be **concise**, without compromising on the principles below.
77 Every element and piece of code on the page should be necessary and relevant 83 Every element and piece of code on the page should be necessary and relevant
(...skipping 610 matching lines...) Expand 10 before | Expand all | Expand 10 after
688 694
689 ### Tests that need to paint, raster, or draw a frame of intermediate output 695 ### Tests that need to paint, raster, or draw a frame of intermediate output
690 696
691 A layout test does not actually draw frames of output until the test exits. 697 A layout test does not actually draw frames of output until the test exits.
692 Tests that need to generate a painted frame can use 698 Tests that need to generate a painted frame can use
693 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a 699 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a
694 frame, then call the passed callback. There is also a library at 700 frame, then call the passed callback. There is also a library at
695 `fast/repaint/resources/text-based-repaint.js` to help with writing paint 701 `fast/repaint/resources/text-based-repaint.js` to help with writing paint
696 invalidation and repaint tests. 702 invalidation and repaint tests.
697 703
698 ## Dump Render Tree (DRT) Tests 704 ## Dump Render Tree (DRT) Tests
pwnall 2016/12/02 02:35:10 Can you please make the name here consistent with
Dirk Pranke 2016/12/02 02:44:54 Will do. It didn't even occur to me to check for m
699 705
700 A Dump Render Tree test renders a web page and produces up to two results, which 706 A Dump Render Tree test renders a web page and produces up to two results, which
701 are compared against baseline files: 707 are compared against baseline files:
702 708
703 * All tests output a textual representation of Blink's 709 * All tests output a textual representation of Blink's
704 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction), 710 [render tree](https://developers.google.com/web/fundamentals/performance/criti cal-rendering-path/render-tree-construction),
705 which is compared against an `-expected.txt` text baseline. 711 which is compared against an `-expected.txt` text baseline.
706 * Some tests also output the image of the rendered page, which is compared 712 * Some tests also output the image of the rendered page, which is compared
707 against an `-expected.png` image baseline, using the same method as pixel 713 against an `-expected.png` image baseline, using the same method as pixel
708 tests. 714 tests.
(...skipping 69 matching lines...) Expand 10 before | Expand all | Expand 10 after
778 * The `http/` directory hosts tests that require an HTTP server (see above). 784 * The `http/` directory hosts tests that require an HTTP server (see above).
779 * The `resources/` subdirectory in every directory contains binary files, such 785 * The `resources/` subdirectory in every directory contains binary files, such
780 as media files, and code that is shared by multiple test files. 786 as media files, and code that is shared by multiple test files.
781 787
782 *** note 788 *** note
783 Some layout tests consist of a minimal HTML page that references a JavaScript 789 Some layout tests consist of a minimal HTML page that references a JavaScript
784 file in `resources/`. Please do not use this pattern for new tests, as it goes 790 file in `resources/`. Please do not use this pattern for new tests, as it goes
785 against the minimality principle. JavaScript and CSS files should only live in 791 against the minimality principle. JavaScript and CSS files should only live in
786 `resources/` if they are shared by at least two test files. 792 `resources/` if they are shared by at least two test files.
787 *** 793 ***
OLDNEW
« no previous file with comments | « no previous file | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698