Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(397)

Side by Side Diff: docs/testing/layout_tests.md

Issue 2492733003: New documentation on writing LayoutTests. (Closed)
Patch Set: New documentation on writing LayoutTests. Created 4 years, 1 month ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
OLDNEW
1 # Layout Tests 1 # Layout Tests
2 2
3 Layout tests are used by Blink to test many components, including but not 3 Layout tests are used by Blink to test many components, including but not
4 limited to layout and rendering. In general, layout tests involve loading pages 4 limited to layout and rendering. In general, layout tests involve loading pages
5 in a test renderer (`content_shell`) and comparing the rendered output or 5 in a test renderer (`content_shell`) and comparing the rendered output or
6 JavaScript output against an expected output file. 6 JavaScript output against an expected output file.
7 7
8 This document covers running and debugging existing layout tests. See the
9 [Writing Layout Tests documentation](./layout_test_writing.md) if you find
10 yourself writing layout tests.
11
8 [TOC] 12 [TOC]
9 13
10 ## Running Layout Tests 14 ## Running Layout Tests
11 15
12 ### Initial Setup 16 ### Initial Setup
13 17
14 Before you can run the layout tests, you need to build the `blink_tests` target 18 Before you can run the layout tests, you need to build the `blink_tests` target
15 to get `content_shell` and all of the other needed binaries. 19 to get `content_shell` and all of the other needed binaries.
16 20
17 ```bash 21 ```bash
(...skipping 37 matching lines...) Expand 10 before | Expand all | Expand 10 after
55 For Android (if your build directory is `out/android`): 59 For Android (if your build directory is `out/android`):
56 60
57 ```bash 61 ```bash
58 python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android 62 python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android
59 ``` 63 ```
60 64
61 Tests marked as `[ Skip ]` in 65 Tests marked as `[ Skip ]` in
62 [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) 66 [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations)
63 won't be run at all, generally because they cause some intractable tool error. 67 won't be run at all, generally because they cause some intractable tool error.
64 To force one of them to be run, either rename that file or specify the skipped 68 To force one of them to be run, either rename that file or specify the skipped
65 test as the only one on the command line (see below). 69 test as the only one on the command line (see below). Read the
70 [Layout Test Expectations documentation](./layout_test_expectations.md) to learn
71 more about TestExpectations and related files.
66 72
67 Note that currently only the tests listed in 73 *** promo
74 Currently only the tests listed in
68 [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests) 75 [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests)
69 are run on the Android bots, since running all layout tests takes too long on 76 are run on the Android bots, since running all layout tests takes too long on
70 Android (and may still have some infrastructure issues). Most developers focus 77 Android (and may still have some infrastructure issues). Most developers focus
71 their Blink testing on Linux. We rely on the fact that the Linux and Android 78 their Blink testing on Linux. We rely on the fact that the Linux and Android
72 behavior is nearly identical for scenarios outside those covered by the smoke 79 behavior is nearly identical for scenarios outside those covered by the smoke
73 tests. 80 tests.
81 ***
74 82
75 To run only some of the tests, specify their directories or filenames as 83 To run only some of the tests, specify their directories or filenames as
76 arguments to `run_webkit_tests.py` relative to the layout test directory 84 arguments to `run_webkit_tests.py` relative to the layout test directory
77 (`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests, 85 (`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests,
78 use: 86 use:
79 87
80 ```bash 88 ```bash
81 Tools/Scripts/run-webkit-tests fast/forms 89 Tools/Scripts/run-webkit-tests fast/forms
82 ``` 90 ```
83 91
(...skipping 88 matching lines...) Expand 10 before | Expand all | Expand 10 after
172 causes the test shell to crash is marked as "crashed", and one that takes longer 180 causes the test shell to crash is marked as "crashed", and one that takes longer
173 than a certain amount of time to complete is aborted and marked as "timed out". 181 than a certain amount of time to complete is aborted and marked as "timed out".
174 A row of dots in the script's output indicates one or more tests that passed. 182 A row of dots in the script's output indicates one or more tests that passed.
175 183
176 ## Test expectations 184 ## Test expectations
177 185
178 The 186 The
179 [TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related 187 [TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related
180 files, including 188 files, including
181 [skia_test_expectations.txt](../../skia/skia_test_expectations.txt)) 189 [skia_test_expectations.txt](../../skia/skia_test_expectations.txt))
182 contains the list of all known layout test failures. See 190 contains the list of all known layout test failures. See the
183 [Test Expectations](./layout_test_expectations.md) 191 [Layout Test Expectations documentation](./layout_test_expectations.md) for more
184 for more on this. 192 on this.
185 193
186 ## Testing Runtime Flags 194 ## Testing Runtime Flags
187 195
188 There are two ways to run layout tests with additional command-line arguments: 196 There are two ways to run layout tests with additional command-line arguments:
189 197
190 * Using `--additional-driver-flag`: 198 * Using `--additional-driver-flag`:
191 199
192 ```bash 200 ```bash
193 run-webkit-tests --additional-driver-flag=--blocking-repaint 201 run-webkit-tests --additional-driver-flag=--blocking-repaint
194 ``` 202 ```
(...skipping 73 matching lines...) Expand 10 before | Expand all | Expand 10 after
268 276
269 * Components: a sub-component of Blink 277 * Components: a sub-component of Blink
270 * OS: **All** (or whichever OS the failure is on) 278 * OS: **All** (or whichever OS the failure is on)
271 * Priority: 2 (1 if it's a crash) 279 * Priority: 2 (1 if it's a crash)
272 * Type: **Bug** 280 * Type: **Bug**
273 * Labels: **Test-Layout** 281 * Labels: **Test-Layout**
274 282
275 You can also use the _Layout Test Failure_ template, which will pre-set these 283 You can also use the _Layout Test Failure_ template, which will pre-set these
276 labels for you. 284 labels for you.
277 285
278 ## Writing Layout Tests
279
280 ### Pixel Tests
281
282 TODO: Write documentation here.
283
284 ### Reference Tests
285
286 TODO: Write documentation here.
287
288 ### Script Tests
289
290 These tests use a JavaScript test harness and test cases written in script to
291 exercise features and make assertions about the behavior. Generally, new tests
292 are written using the [testharness.js](https://github.com/w3c/testharness.js/)
293 test harness, which is also heavily used in the cross-vendor
294 [web-platform-tests](https://github.com/w3c/web-platform-tests) project. Tests
295 written with testharness.js generally look something like the following:
296
297 ```html
298 <!DOCTYPE html>
299 <script src="/resources/testharness.js"></script>
300 <script src="/resources/testharnessreport.js"></script>
301 <script>
302 test(t => {
303 var x = true;
304 assert_true(x);
305 }, "Truth is true.");
306 </script>
307 ```
308
309 Many older tests are written using the **js-test**
310 (`LayoutTests/resources/js-test.js`) test harness. This harness is
311 **deprecated**, and should not be used for new tests. The tests call
312 `testRunner.dumpAsText()` to signal that the page content should be dumped and
313 compared against an \*-expected.txt file, and optionally
314 `testRunner.waitUntilDone()` and `testRunner.notifyDone()` for asynchronous
315 tests.
316
317 ### Tests that use a HTTP Server
318
319 By default, tests are loaded as if via `file:` URLs. Some web platform features
320 require tests served via HTTP or HTTPS, for example relative paths (`src=/foo`)
321 or features restricted to secure protocols.
322
323 HTTP tests are those tests that are under `LayoutTests/http/tests` (or virtual
324 variants). Use a locally running HTTP server (Apache) to run. Tests are served
325 off of ports 8000, 8080 for HTTP and 8443 for HTTPS. If you run the tests using
326 `run-webkit-tests`, the server will be started automatically.To run the server
327 manually to reproduce or debug a failure:
328
329 ```bash
330 cd src/third_party/WebKit/Tools/Scripts
331 run-blink-httpd start
332 ```
333
334 The layout tests will be served from `http://127.0.0.1:8000`. For example, to
335 run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
336 navigate to
337 `http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
338 tests will behave differently if you go to 127.0.0.1 instead of localhost, so
339 use 127.0.0.1.
340
341 To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
342 or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.
343
344 The test server sets up an alias to `LayoutTests/resources` directory. In HTTP
345 tests, you can access the testing framework at e.g.
346 `src="/js-test-resources/js-test.js"`.
347
348 ### Writing tests that need to paint, raster, or draw a frame of intermediate ou tput
349
350 A layout test does not actually draw frames of output until the test exits. If
351 it is required to generate a painted frame, then use
352 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a
353 frame, then call the passed callback. There is also a library at
354 `fast/repaint/resources/text-based-repaint.js` to help with writing paint
355 invalidation and repaint tests.
356
357 #### Layout test support for `testRunner`
358
359 Some layout tests rely on the testRunner object to expose configuration for
360 mocking the platform. This is provided in content_shell, here's a UML diagram of
361 testRunner bindings configuring platform implementation:
362
363 [![UML of testRunner bindings configuring platform implementation](https://docs. google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/sv g?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google .com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
364
365 [Writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1 O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit)
366
367 ## Debugging Layout Tests 286 ## Debugging Layout Tests
368 287
369 After the layout tests run, you should get a summary of tests that pass or fail. 288 After the layout tests run, you should get a summary of tests that pass or fail.
370 If something fails unexpectedly (a new regression), you will get a content_shell 289 If something fails unexpectedly (a new regression), you will get a content_shell
371 window with a summary of the unexpected failures. Or you might have a failing 290 window with a summary of the unexpected failures. Or you might have a failing
372 test in mind to investigate. In any case, here are some steps and tips for 291 test in mind to investigate. In any case, here are some steps and tips for
373 finding the problem. 292 finding the problem.
374 293
375 * Take a look at the result. Sometimes tests just need to be rebaselined (see 294 * Take a look at the result. Sometimes tests just need to be rebaselined (see
376 below) to account for changes introduced in your patch. 295 below) to account for changes introduced in your patch.
(...skipping 179 matching lines...) Expand 10 before | Expand all | Expand 10 after
556 See 475 See
557 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue s/list?can=2&q=component%3ABlink%3EInfra) 476 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue s/list?can=2&q=component%3ABlink%3EInfra)
558 for issues related to Blink tools, include the layout test runner. 477 for issues related to Blink tools, include the layout test runner.
559 478
560 * Windows and Linux: Do not copy and paste while the layout tests are running, 479 * Windows and Linux: Do not copy and paste while the layout tests are running,
561 as it may interfere with the editing/pasteboard and other clipboard-related 480 as it may interfere with the editing/pasteboard and other clipboard-related
562 tests. (Mac tests swizzle NSClipboard to avoid any conflicts). 481 tests. (Mac tests swizzle NSClipboard to avoid any conflicts).
563 * If QuickTime is not installed, the plugin tests 482 * If QuickTime is not installed, the plugin tests
564 `fast/dom/object-embed-plugin-scripting.html` and 483 `fast/dom/object-embed-plugin-scripting.html` and
565 `plugins/embed-attributes-setting.html` are expected to fail. 484 `plugins/embed-attributes-setting.html` are expected to fail.
OLDNEW
« docs/testing/layout_test_writing.md ('K') | « docs/testing/layout_test_writing.md ('k') | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698