| OLD | NEW |
| 1 # Layout Tests | 1 # Layout Tests |
| 2 | 2 |
| 3 Layout tests are used by Blink to test many components, including but not | 3 Layout tests are used by Blink to test many components, including but not |
| 4 limited to layout and rendering. In general, layout tests involve loading pages | 4 limited to layout and rendering. In general, layout tests involve loading pages |
| 5 in a test renderer (`content_shell`) and comparing the rendered output or | 5 in a test renderer (`content_shell`) and comparing the rendered output or |
| 6 JavaScript output against an expected output file. | 6 JavaScript output against an expected output file. |
| 7 | 7 |
| 8 This document covers running and debugging existing layout tests. See the |
| 9 [Writing Layout Tests documentation](./layout_test_writing.md) if you find |
| 10 yourself writing layout tests. |
| 11 |
| 8 [TOC] | 12 [TOC] |
| 9 | 13 |
| 10 ## Running Layout Tests | 14 ## Running Layout Tests |
| 11 | 15 |
| 12 ### Initial Setup | 16 ### Initial Setup |
| 13 | 17 |
| 14 Before you can run the layout tests, you need to build the `blink_tests` target | 18 Before you can run the layout tests, you need to build the `blink_tests` target |
| 15 to get `content_shell` and all of the other needed binaries. | 19 to get `content_shell` and all of the other needed binaries. |
| 16 | 20 |
| 17 ```bash | 21 ```bash |
| (...skipping 37 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 55 For Android (if your build directory is `out/android`): | 59 For Android (if your build directory is `out/android`): |
| 56 | 60 |
| 57 ```bash | 61 ```bash |
| 58 python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android | 62 python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android |
| 59 ``` | 63 ``` |
| 60 | 64 |
| 61 Tests marked as `[ Skip ]` in | 65 Tests marked as `[ Skip ]` in |
| 62 [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) | 66 [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) |
| 63 won't be run at all, generally because they cause some intractable tool error. | 67 won't be run at all, generally because they cause some intractable tool error. |
| 64 To force one of them to be run, either rename that file or specify the skipped | 68 To force one of them to be run, either rename that file or specify the skipped |
| 65 test as the only one on the command line (see below). | 69 test as the only one on the command line (see below). Read the |
| 70 [Layout Test Expectations documentation](./layout_test_expectations.md) to learn |
| 71 more about TestExpectations and related files. |
| 66 | 72 |
| 67 Note that currently only the tests listed in | 73 *** promo |
| 74 Currently only the tests listed in |
| 68 [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests) | 75 [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests) |
| 69 are run on the Android bots, since running all layout tests takes too long on | 76 are run on the Android bots, since running all layout tests takes too long on |
| 70 Android (and may still have some infrastructure issues). Most developers focus | 77 Android (and may still have some infrastructure issues). Most developers focus |
| 71 their Blink testing on Linux. We rely on the fact that the Linux and Android | 78 their Blink testing on Linux. We rely on the fact that the Linux and Android |
| 72 behavior is nearly identical for scenarios outside those covered by the smoke | 79 behavior is nearly identical for scenarios outside those covered by the smoke |
| 73 tests. | 80 tests. |
| 81 *** |
| 74 | 82 |
| 75 To run only some of the tests, specify their directories or filenames as | 83 To run only some of the tests, specify their directories or filenames as |
| 76 arguments to `run_webkit_tests.py` relative to the layout test directory | 84 arguments to `run_webkit_tests.py` relative to the layout test directory |
| 77 (`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests, | 85 (`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests, |
| 78 use: | 86 use: |
| 79 | 87 |
| 80 ```bash | 88 ```bash |
| 81 Tools/Scripts/run-webkit-tests fast/forms | 89 Tools/Scripts/run-webkit-tests fast/forms |
| 82 ``` | 90 ``` |
| 83 | 91 |
| (...skipping 88 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 172 causes the test shell to crash is marked as "crashed", and one that takes longer | 180 causes the test shell to crash is marked as "crashed", and one that takes longer |
| 173 than a certain amount of time to complete is aborted and marked as "timed out". | 181 than a certain amount of time to complete is aborted and marked as "timed out". |
| 174 A row of dots in the script's output indicates one or more tests that passed. | 182 A row of dots in the script's output indicates one or more tests that passed. |
| 175 | 183 |
| 176 ## Test expectations | 184 ## Test expectations |
| 177 | 185 |
| 178 The | 186 The |
| 179 [TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related | 187 [TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related |
| 180 files, including | 188 files, including |
| 181 [skia_test_expectations.txt](../../skia/skia_test_expectations.txt)) | 189 [skia_test_expectations.txt](../../skia/skia_test_expectations.txt)) |
| 182 contains the list of all known layout test failures. See | 190 contains the list of all known layout test failures. See the |
| 183 [Test Expectations](./layout_test_expectations.md) | 191 [Layout Test Expectations documentation](./layout_test_expectations.md) for more |
| 184 for more on this. | 192 on this. |
| 185 | 193 |
| 186 ## Testing Runtime Flags | 194 ## Testing Runtime Flags |
| 187 | 195 |
| 188 There are two ways to run layout tests with additional command-line arguments: | 196 There are two ways to run layout tests with additional command-line arguments: |
| 189 | 197 |
| 190 * Using `--additional-driver-flag`: | 198 * Using `--additional-driver-flag`: |
| 191 | 199 |
| 192 ```bash | 200 ```bash |
| 193 run-webkit-tests --additional-driver-flag=--blocking-repaint | 201 run-webkit-tests --additional-driver-flag=--blocking-repaint |
| 194 ``` | 202 ``` |
| (...skipping 73 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 268 | 276 |
| 269 * Components: a sub-component of Blink | 277 * Components: a sub-component of Blink |
| 270 * OS: **All** (or whichever OS the failure is on) | 278 * OS: **All** (or whichever OS the failure is on) |
| 271 * Priority: 2 (1 if it's a crash) | 279 * Priority: 2 (1 if it's a crash) |
| 272 * Type: **Bug** | 280 * Type: **Bug** |
| 273 * Labels: **Test-Layout** | 281 * Labels: **Test-Layout** |
| 274 | 282 |
| 275 You can also use the _Layout Test Failure_ template, which will pre-set these | 283 You can also use the _Layout Test Failure_ template, which will pre-set these |
| 276 labels for you. | 284 labels for you. |
| 277 | 285 |
| 278 ## Writing Layout Tests | |
| 279 | |
| 280 ### Pixel Tests | |
| 281 | |
| 282 TODO: Write documentation here. | |
| 283 | |
| 284 ### Reference Tests | |
| 285 | |
| 286 TODO: Write documentation here. | |
| 287 | |
| 288 ### Script Tests | |
| 289 | |
| 290 These tests use a JavaScript test harness and test cases written in script to | |
| 291 exercise features and make assertions about the behavior. Generally, new tests | |
| 292 are written using the [testharness.js](https://github.com/w3c/testharness.js/) | |
| 293 test harness, which is also heavily used in the cross-vendor | |
| 294 [web-platform-tests](https://github.com/w3c/web-platform-tests) project. Tests | |
| 295 written with testharness.js generally look something like the following: | |
| 296 | |
| 297 ```html | |
| 298 <!DOCTYPE html> | |
| 299 <script src="/resources/testharness.js"></script> | |
| 300 <script src="/resources/testharnessreport.js"></script> | |
| 301 <script> | |
| 302 test(t => { | |
| 303 var x = true; | |
| 304 assert_true(x); | |
| 305 }, "Truth is true."); | |
| 306 </script> | |
| 307 ``` | |
| 308 | |
| 309 Many older tests are written using the **js-test** | |
| 310 (`LayoutTests/resources/js-test.js`) test harness. This harness is | |
| 311 **deprecated**, and should not be used for new tests. The tests call | |
| 312 `testRunner.dumpAsText()` to signal that the page content should be dumped and | |
| 313 compared against an \*-expected.txt file, and optionally | |
| 314 `testRunner.waitUntilDone()` and `testRunner.notifyDone()` for asynchronous | |
| 315 tests. | |
| 316 | |
| 317 ### Tests that use a HTTP Server | |
| 318 | |
| 319 By default, tests are loaded as if via `file:` URLs. Some web platform features | |
| 320 require tests served via HTTP or HTTPS, for example relative paths (`src=/foo`) | |
| 321 or features restricted to secure protocols. | |
| 322 | |
| 323 HTTP tests are those tests that are under `LayoutTests/http/tests` (or virtual | |
| 324 variants). Use a locally running HTTP server (Apache) to run. Tests are served | |
| 325 off of ports 8000, 8080 for HTTP and 8443 for HTTPS. If you run the tests using | |
| 326 `run-webkit-tests`, the server will be started automatically.To run the server | |
| 327 manually to reproduce or debug a failure: | |
| 328 | |
| 329 ```bash | |
| 330 cd src/third_party/WebKit/Tools/Scripts | |
| 331 run-blink-httpd start | |
| 332 ``` | |
| 333 | |
| 334 The layout tests will be served from `http://127.0.0.1:8000`. For example, to | |
| 335 run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`, | |
| 336 navigate to | |
| 337 `http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some | |
| 338 tests will behave differently if you go to 127.0.0.1 instead of localhost, so | |
| 339 use 127.0.0.1. | |
| 340 | |
| 341 To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill` | |
| 342 or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS. | |
| 343 | |
| 344 The test server sets up an alias to `LayoutTests/resources` directory. In HTTP | |
| 345 tests, you can access the testing framework at e.g. | |
| 346 `src="/js-test-resources/js-test.js"`. | |
| 347 | |
| 348 ### Writing tests that need to paint, raster, or draw a frame of intermediate ou
tput | |
| 349 | |
| 350 A layout test does not actually draw frames of output until the test exits. If | |
| 351 it is required to generate a painted frame, then use | |
| 352 `window.testRunner.displayAsyncThen`, which will run the machinery to put up a | |
| 353 frame, then call the passed callback. There is also a library at | |
| 354 `fast/repaint/resources/text-based-repaint.js` to help with writing paint | |
| 355 invalidation and repaint tests. | |
| 356 | |
| 357 #### Layout test support for `testRunner` | |
| 358 | |
| 359 Some layout tests rely on the testRunner object to expose configuration for | |
| 360 mocking the platform. This is provided in content_shell, here's a UML diagram of | |
| 361 testRunner bindings configuring platform implementation: | |
| 362 | |
| 363 [](https://docs.google
.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) | |
| 364 | |
| 365 [Writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1
O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit) | |
| 366 | |
| 367 ## Debugging Layout Tests | 286 ## Debugging Layout Tests |
| 368 | 287 |
| 369 After the layout tests run, you should get a summary of tests that pass or fail. | 288 After the layout tests run, you should get a summary of tests that pass or fail. |
| 370 If something fails unexpectedly (a new regression), you will get a content_shell | 289 If something fails unexpectedly (a new regression), you will get a content_shell |
| 371 window with a summary of the unexpected failures. Or you might have a failing | 290 window with a summary of the unexpected failures. Or you might have a failing |
| 372 test in mind to investigate. In any case, here are some steps and tips for | 291 test in mind to investigate. In any case, here are some steps and tips for |
| 373 finding the problem. | 292 finding the problem. |
| 374 | 293 |
| 375 * Take a look at the result. Sometimes tests just need to be rebaselined (see | 294 * Take a look at the result. Sometimes tests just need to be rebaselined (see |
| 376 below) to account for changes introduced in your patch. | 295 below) to account for changes introduced in your patch. |
| (...skipping 179 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 556 See | 475 See |
| 557 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue
s/list?can=2&q=component%3ABlink%3EInfra) | 476 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue
s/list?can=2&q=component%3ABlink%3EInfra) |
| 558 for issues related to Blink tools, include the layout test runner. | 477 for issues related to Blink tools, include the layout test runner. |
| 559 | 478 |
| 560 * Windows and Linux: Do not copy and paste while the layout tests are running, | 479 * Windows and Linux: Do not copy and paste while the layout tests are running, |
| 561 as it may interfere with the editing/pasteboard and other clipboard-related | 480 as it may interfere with the editing/pasteboard and other clipboard-related |
| 562 tests. (Mac tests swizzle NSClipboard to avoid any conflicts). | 481 tests. (Mac tests swizzle NSClipboard to avoid any conflicts). |
| 563 * If QuickTime is not installed, the plugin tests | 482 * If QuickTime is not installed, the plugin tests |
| 564 `fast/dom/object-embed-plugin-scripting.html` and | 483 `fast/dom/object-embed-plugin-scripting.html` and |
| 565 `plugins/embed-attributes-setting.html` are expected to fail. | 484 `plugins/embed-attributes-setting.html` are expected to fail. |
| OLD | NEW |