OLD | NEW |
1 # Layout Tests | 1 # Layout Tests |
2 | 2 |
3 Layout tests are used by Blink to test many components, including but not | 3 Layout tests are used by Blink to test many components, including but not |
4 limited to layout and rendering. In general, layout tests involve loading pages | 4 limited to layout and rendering. In general, layout tests involve loading pages |
5 in a test renderer (`content_shell`) and comparing the rendered output or | 5 in a test renderer (`content_shell`) and comparing the rendered output or |
6 JavaScript output against an expected output file. | 6 JavaScript output against an expected output file. |
7 | 7 |
| 8 This document covers running and debugging existing layout tests. See the |
| 9 [Writing Layout Tests documentation](./layout_test_writing.md) if you find |
| 10 yourself writing layout tests. |
| 11 |
8 [TOC] | 12 [TOC] |
9 | 13 |
10 ## Running Layout Tests | 14 ## Running Layout Tests |
11 | 15 |
12 ### Initial Setup | 16 ### Initial Setup |
13 | 17 |
14 Before you can run the layout tests, you need to build the `blink_tests` target | 18 Before you can run the layout tests, you need to build the `blink_tests` target |
15 to get `content_shell` and all of the other needed binaries. | 19 to get `content_shell` and all of the other needed binaries. |
16 | 20 |
17 ```bash | 21 ```bash |
(...skipping 37 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
55 For Android (if your build directory is `out/android`): | 59 For Android (if your build directory is `out/android`): |
56 | 60 |
57 ```bash | 61 ```bash |
58 python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android | 62 python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android |
59 ``` | 63 ``` |
60 | 64 |
61 Tests marked as `[ Skip ]` in | 65 Tests marked as `[ Skip ]` in |
62 [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) | 66 [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) |
63 won't be run at all, generally because they cause some intractable tool error. | 67 won't be run at all, generally because they cause some intractable tool error. |
64 To force one of them to be run, either rename that file or specify the skipped | 68 To force one of them to be run, either rename that file or specify the skipped |
65 test as the only one on the command line (see below). | 69 test as the only one on the command line (see below). Read the |
| 70 [Layout Test Expectations documentation](./layout_test_expectations.md) to learn |
| 71 more about TestExpectations and related files. |
66 | 72 |
67 Note that currently only the tests listed in | 73 *** promo |
| 74 Currently only the tests listed in |
68 [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests) | 75 [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests) |
69 are run on the Android bots, since running all layout tests takes too long on | 76 are run on the Android bots, since running all layout tests takes too long on |
70 Android (and may still have some infrastructure issues). Most developers focus | 77 Android (and may still have some infrastructure issues). Most developers focus |
71 their Blink testing on Linux. We rely on the fact that the Linux and Android | 78 their Blink testing on Linux. We rely on the fact that the Linux and Android |
72 behavior is nearly identical for scenarios outside those covered by the smoke | 79 behavior is nearly identical for scenarios outside those covered by the smoke |
73 tests. | 80 tests. |
| 81 *** |
74 | 82 |
75 To run only some of the tests, specify their directories or filenames as | 83 To run only some of the tests, specify their directories or filenames as |
76 arguments to `run_webkit_tests.py` relative to the layout test directory | 84 arguments to `run_webkit_tests.py` relative to the layout test directory |
77 (`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests, | 85 (`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests, |
78 use: | 86 use: |
79 | 87 |
80 ```bash | 88 ```bash |
81 Tools/Scripts/run-webkit-tests fast/forms | 89 Tools/Scripts/run-webkit-tests fast/forms |
82 ``` | 90 ``` |
83 | 91 |
(...skipping 88 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
172 causes the test shell to crash is marked as "crashed", and one that takes longer | 180 causes the test shell to crash is marked as "crashed", and one that takes longer |
173 than a certain amount of time to complete is aborted and marked as "timed out". | 181 than a certain amount of time to complete is aborted and marked as "timed out". |
174 A row of dots in the script's output indicates one or more tests that passed. | 182 A row of dots in the script's output indicates one or more tests that passed. |
175 | 183 |
176 ## Test expectations | 184 ## Test expectations |
177 | 185 |
178 The | 186 The |
179 [TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related | 187 [TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related |
180 files, including | 188 files, including |
181 [skia_test_expectations.txt](../../skia/skia_test_expectations.txt)) | 189 [skia_test_expectations.txt](../../skia/skia_test_expectations.txt)) |
182 contains the list of all known layout test failures. See | 190 contains the list of all known layout test failures. See the |
183 [Test Expectations](./layout_test_expectations.md) | 191 [Layout Test Expectations documentation](./layout_test_expectations.md) for more |
184 for more on this. | 192 on this. |
185 | 193 |
186 ## Testing Runtime Flags | 194 ## Testing Runtime Flags |
187 | 195 |
188 There are two ways to run layout tests with additional command-line arguments: | 196 There are two ways to run layout tests with additional command-line arguments: |
189 | 197 |
190 * Using `--additional-driver-flag`: | 198 * Using `--additional-driver-flag`: |
191 | 199 |
192 ```bash | 200 ```bash |
193 run-webkit-tests --additional-driver-flag=--blocking-repaint | 201 run-webkit-tests --additional-driver-flag=--blocking-repaint |
194 ``` | 202 ``` |
(...skipping 73 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
268 | 276 |
269 * Components: a sub-component of Blink | 277 * Components: a sub-component of Blink |
270 * OS: **All** (or whichever OS the failure is on) | 278 * OS: **All** (or whichever OS the failure is on) |
271 * Priority: 2 (1 if it's a crash) | 279 * Priority: 2 (1 if it's a crash) |
272 * Type: **Bug** | 280 * Type: **Bug** |
273 * Labels: **Test-Layout** | 281 * Labels: **Test-Layout** |
274 | 282 |
275 You can also use the _Layout Test Failure_ template, which will pre-set these | 283 You can also use the _Layout Test Failure_ template, which will pre-set these |
276 labels for you. | 284 labels for you. |
277 | 285 |
278 ## Writing Layout Tests | |
279 | |
280 ### Pixel Tests | |
281 | |
282 TODO: Write documentation here. | |
283 | |
284 ### Reference Tests | |
285 | |
286 TODO: Write documentation here. | |
287 | |
288 ### Script Tests | |
289 | |
290 These tests use a JavaScript test harness and test cases written in script to | |
291 exercise features and make assertions about the behavior. Generally, new tests | |
292 are written using the [testharness.js](https://github.com/w3c/testharness.js/) | |
293 test harness, which is also heavily used in the cross-vendor | |
294 [web-platform-tests](https://github.com/w3c/web-platform-tests) project. Tests | |
295 written with testharness.js generally look something like the following: | |
296 | |
297 ```html | |
298 <!DOCTYPE html> | |
299 <script src="/resources/testharness.js"></script> | |
300 <script src="/resources/testharnessreport.js"></script> | |
301 <script> | |
302 test(t => { | |
303 var x = true; | |
304 assert_true(x); | |
305 }, "Truth is true."); | |
306 </script> | |
307 ``` | |
308 | |
309 Many older tests are written using the **js-test** | |
310 (`LayoutTests/resources/js-test.js`) test harness. This harness is | |
311 **deprecated**, and should not be used for new tests. The tests call | |
312 `testRunner.dumpAsText()` to signal that the page content should be dumped and | |
313 compared against an \*-expected.txt file, and optionally | |
314 `testRunner.waitUntilDone()` and `testRunner.notifyDone()` for asynchronous | |
315 tests. | |
316 | |
317 ### Tests that use a HTTP Server | |
318 | |
319 By default, tests are loaded as if via `file:` URLs. Some web platform features | |
320 require tests served via HTTP or HTTPS, for example relative paths (`src=/foo`) | |
321 or features restricted to secure protocols. | |
322 | |
323 HTTP tests are those tests that are under `LayoutTests/http/tests` (or virtual | |
324 variants). Use a locally running HTTP server (Apache) to run. Tests are served | |
325 off of ports 8000, 8080 for HTTP and 8443 for HTTPS. If you run the tests using | |
326 `run-webkit-tests`, the server will be started automatically.To run the server | |
327 manually to reproduce or debug a failure: | |
328 | |
329 ```bash | |
330 cd src/third_party/WebKit/Tools/Scripts | |
331 run-blink-httpd start | |
332 ``` | |
333 | |
334 The layout tests will be served from `http://127.0.0.1:8000`. For example, to | |
335 run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`, | |
336 navigate to | |
337 `http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some | |
338 tests will behave differently if you go to 127.0.0.1 instead of localhost, so | |
339 use 127.0.0.1. | |
340 | |
341 To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill` | |
342 or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS. | |
343 | |
344 The test server sets up an alias to `LayoutTests/resources` directory. In HTTP | |
345 tests, you can access the testing framework at e.g. | |
346 `src="/js-test-resources/js-test.js"`. | |
347 | |
348 ### Writing tests that need to paint, raster, or draw a frame of intermediate ou
tput | |
349 | |
350 A layout test does not actually draw frames of output until the test exits. If | |
351 it is required to generate a painted frame, then use | |
352 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a | |
353 frame, then call the passed callback. There is also a library at | |
354 `fast/repaint/resources/text-based-repaint.js` to help with writing paint | |
355 invalidation and repaint tests. | |
356 | |
357 #### Layout test support for `testRunner` | |
358 | |
359 Some layout tests rely on the testRunner object to expose configuration for | |
360 mocking the platform. This is provided in content_shell, here's a UML diagram of | |
361 testRunner bindings configuring platform implementation: | |
362 | |
363 [](https://docs.google
.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) | |
364 | |
365 [Writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1
O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit) | |
366 | |
367 ## Debugging Layout Tests | 286 ## Debugging Layout Tests |
368 | 287 |
369 After the layout tests run, you should get a summary of tests that pass or fail. | 288 After the layout tests run, you should get a summary of tests that pass or fail. |
370 If something fails unexpectedly (a new regression), you will get a content_shell | 289 If something fails unexpectedly (a new regression), you will get a content_shell |
371 window with a summary of the unexpected failures. Or you might have a failing | 290 window with a summary of the unexpected failures. Or you might have a failing |
372 test in mind to investigate. In any case, here are some steps and tips for | 291 test in mind to investigate. In any case, here are some steps and tips for |
373 finding the problem. | 292 finding the problem. |
374 | 293 |
375 * Take a look at the result. Sometimes tests just need to be rebaselined (see | 294 * Take a look at the result. Sometimes tests just need to be rebaselined (see |
376 below) to account for changes introduced in your patch. | 295 below) to account for changes introduced in your patch. |
(...skipping 179 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
556 See | 475 See |
557 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue
s/list?can=2&q=component%3ABlink%3EInfra) | 476 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue
s/list?can=2&q=component%3ABlink%3EInfra) |
558 for issues related to Blink tools, include the layout test runner. | 477 for issues related to Blink tools, include the layout test runner. |
559 | 478 |
560 * Windows and Linux: Do not copy and paste while the layout tests are running, | 479 * Windows and Linux: Do not copy and paste while the layout tests are running, |
561 as it may interfere with the editing/pasteboard and other clipboard-related | 480 as it may interfere with the editing/pasteboard and other clipboard-related |
562 tests. (Mac tests swizzle NSClipboard to avoid any conflicts). | 481 tests. (Mac tests swizzle NSClipboard to avoid any conflicts). |
563 * If QuickTime is not installed, the plugin tests | 482 * If QuickTime is not installed, the plugin tests |
564 `fast/dom/object-embed-plugin-scripting.html` and | 483 `fast/dom/object-embed-plugin-scripting.html` and |
565 `plugins/embed-attributes-setting.html` are expected to fail. | 484 `plugins/embed-attributes-setting.html` are expected to fail. |
OLD | NEW |