OLD | NEW |
1 # Layout Tests | 1 # Layout Tests |
2 | 2 |
3 Layout tests are used by Blink to test many components, including but not | 3 Layout tests are used by Blink to test many components, including but not |
4 limited to layout and rendering. In general, layout tests involve loading pages | 4 limited to layout and rendering. In general, layout tests involve loading pages |
5 in a test renderer (`content_shell`) and comparing the rendered output or | 5 in a test renderer (`content_shell`) and comparing the rendered output or |
6 JavaScript output against an expected output file. | 6 JavaScript output against an expected output file. |
7 | 7 |
8 This document covers running and debugging existing layout tests. See the | 8 This document covers running and debugging existing layout tests. See the |
9 [Writing Layout Tests documentation](./writing_layout_tests.md) if you find | 9 [Writing Layout Tests documentation](./writing_layout_tests.md) if you find |
10 yourself writing layout tests. | 10 yourself writing layout tests. |
(...skipping 166 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
177 *** | 177 *** |
178 | 178 |
179 A test that runs but produces the wrong output is marked as "failed", one that | 179 A test that runs but produces the wrong output is marked as "failed", one that |
180 causes the test shell to crash is marked as "crashed", and one that takes longer | 180 causes the test shell to crash is marked as "crashed", and one that takes longer |
181 than a certain amount of time to complete is aborted and marked as "timed out". | 181 than a certain amount of time to complete is aborted and marked as "timed out". |
182 A row of dots in the script's output indicates one or more tests that passed. | 182 A row of dots in the script's output indicates one or more tests that passed. |
183 | 183 |
184 ## Test expectations | 184 ## Test expectations |
185 | 185 |
186 The | 186 The |
187 [TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related | 187 [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) file (
and related |
188 files, including | 188 files) contains the list of all known layout test failures. See the |
189 [skia_test_expectations.txt](../../skia/skia_test_expectations.txt)) | |
190 contains the list of all known layout test failures. See the | |
191 [Layout Test Expectations documentation](./layout_test_expectations.md) for more | 189 [Layout Test Expectations documentation](./layout_test_expectations.md) for more |
192 on this. | 190 on this. |
193 | 191 |
194 ## Testing Runtime Flags | 192 ## Testing Runtime Flags |
195 | 193 |
196 There are two ways to run layout tests with additional command-line arguments: | 194 There are two ways to run layout tests with additional command-line arguments: |
197 | 195 |
198 * Using `--additional-driver-flag`: | 196 * Using `--additional-driver-flag`: |
199 | 197 |
200 ```bash | 198 ```bash |
201 run-webkit-tests --additional-driver-flag=--blocking-repaint | 199 run-webkit-tests --additional-driver-flag=--blocking-repaint |
202 ``` | 200 ``` |
203 | 201 |
204 This tells the test harness to pass `--blocking-repaint` to the | 202 This tells the test harness to pass `--blocking-repaint` to the |
205 content_shell binary. | 203 content_shell binary. |
206 | 204 |
207 It will also look for flag-specific expectations in | 205 It will also look for flag-specific expectations in |
208 `LayoutTests/FlagExpectations/blocking-repaint`, if this file exists. The | 206 `LayoutTests/FlagExpectations/blocking-repaint`, if this file exists. The |
209 suppressions in this file override the main TestExpectations file. | 207 suppressions in this file override the main TestExpectations file. |
210 | 208 |
211 * Using a *virtual test suite* defined in | 209 * Using a *virtual test suite* defined in |
212 [LayoutTests/VirtualTestSuites](https://code.google.com/p/chromium/codesearch#
chromium/src/third_party/WebKit/LayoutTests/VirtualTestSuites). | 210 [LayoutTests/VirtualTestSuites](../../third_party/WebKit/LayoutTests/VirtualTe
stSuites). |
213 A virtual test suite runs a subset of layout tests under a specific path with | 211 A virtual test suite runs a subset of layout tests under a specific path with |
214 additional flags. For example, you could test a (hypothetical) new mode for | 212 additional flags. For example, you could test a (hypothetical) new mode for |
215 repainting using the following virtual test suite: | 213 repainting using the following virtual test suite: |
216 | 214 |
217 ```json | 215 ```json |
218 { | 216 { |
219 "prefix": "blocking_repaint", | 217 "prefix": "blocking_repaint", |
220 "base": "fast/repaint", | 218 "base": "fast/repaint", |
221 "args": ["--blocking-repaint"], | 219 "args": ["--blocking-repaint"], |
222 } | 220 } |
(...skipping 94 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
317 produce a page including links to the expected result, actual result, and | 315 produce a page including links to the expected result, actual result, and |
318 diff. | 316 diff. |
319 * Add the `--sources` option to `run_webkit_tests.py` to see exactly which | 317 * Add the `--sources` option to `run_webkit_tests.py` to see exactly which |
320 expected result it's comparing to (a file next to the test, something in | 318 expected result it's comparing to (a file next to the test, something in |
321 platform/mac/, something in platform/chromium-win/, etc.) | 319 platform/mac/, something in platform/chromium-win/, etc.) |
322 * If you're still sure it's correct, rebaseline the test (see below). | 320 * If you're still sure it's correct, rebaseline the test (see below). |
323 Otherwise... | 321 Otherwise... |
324 * If you're lucky, your test is one that runs properly when you navigate to it | 322 * If you're lucky, your test is one that runs properly when you navigate to it |
325 in content_shell normally. In that case, build the Debug content_shell | 323 in content_shell normally. In that case, build the Debug content_shell |
326 project, fire it up in your favorite debugger, and load the test file either | 324 project, fire it up in your favorite debugger, and load the test file either |
327 from a file:// URL. | 325 from a `file:` URL. |
328 * You'll probably be starting and stopping the content_shell a lot. In VS, | 326 * You'll probably be starting and stopping the content_shell a lot. In VS, |
329 to save navigating to the test every time, you can set the URL to your | 327 to save navigating to the test every time, you can set the URL to your |
330 test (file: or http:) as the command argument in the Debugging section of | 328 test (`file:` or `http:`) as the command argument in the Debugging section
of |
331 the content_shell project Properties. | 329 the content_shell project Properties. |
332 * If your test contains a JS call, DOM manipulation, or other distinctive | 330 * If your test contains a JS call, DOM manipulation, or other distinctive |
333 piece of code that you think is failing, search for that in the Chrome | 331 piece of code that you think is failing, search for that in the Chrome |
334 solution. That's a good place to put a starting breakpoint to start | 332 solution. That's a good place to put a starting breakpoint to start |
335 tracking down the issue. | 333 tracking down the issue. |
336 * Otherwise, you're running in a standard message loop just like in Chrome. | 334 * Otherwise, you're running in a standard message loop just like in Chrome. |
337 If you have no other information, set a breakpoint on page load. | 335 If you have no other information, set a breakpoint on page load. |
338 * If your test only works in full layout-test mode, or if you find it simpler to | 336 * If your test only works in full layout-test mode, or if you find it simpler to |
339 debug without all the overhead of an interactive session, start the | 337 debug without all the overhead of an interactive session, start the |
340 content_shell with the command-line flag `--run-layout-test`, followed by the | 338 content_shell with the command-line flag `--run-layout-test`, followed by the |
341 URL (file: or http:) to your test. More information about running layout tests | 339 URL (`file:` or `http:`) to your test. More information about running layout t
ests |
342 in content_shell can be found [here](./layout_tests_in_content_shell.md). | 340 in content_shell can be found [here](./layout_tests_in_content_shell.md). |
343 * In VS, you can do this in the Debugging section of the content_shell | 341 * In VS, you can do this in the Debugging section of the content_shell |
344 project Properties. | 342 project Properties. |
345 * Now you're running with exactly the same API, theme, and other setup that | 343 * Now you're running with exactly the same API, theme, and other setup that |
346 the layout tests use. | 344 the layout tests use. |
347 * Again, if your test contains a JS call, DOM manipulation, or other | 345 * Again, if your test contains a JS call, DOM manipulation, or other |
348 distinctive piece of code that you think is failing, search for that in | 346 distinctive piece of code that you think is failing, search for that in |
349 the Chrome solution. That's a good place to put a starting breakpoint to | 347 the Chrome solution. That's a good place to put a starting breakpoint to |
350 start tracking down the issue. | 348 start tracking down the issue. |
351 * If you can't find any better place to set a breakpoint, start at the | 349 * If you can't find any better place to set a breakpoint, start at the |
(...skipping 65 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
417 `blink/tools/run_layout_tests.sh | 415 `blink/tools/run_layout_tests.sh |
418 --additional_driver_flag='--remote-debugging-port=9222' | 416 --additional_driver_flag='--remote-debugging-port=9222' |
419 --time-out-ms=6000000` | 417 --time-out-ms=6000000` |
420 * Option B) If you need to debug an http/tests/inspector test, start httpd | 418 * Option B) If you need to debug an http/tests/inspector test, start httpd |
421 as described above. Then, run content_shell: | 419 as described above. Then, run content_shell: |
422 `out/Default/content_shell --remote-debugging-port=9222 --run-layout-test | 420 `out/Default/content_shell --remote-debugging-port=9222 --run-layout-test |
423 http://127.0.0.1:8000/path/to/test.html` | 421 http://127.0.0.1:8000/path/to/test.html` |
424 * Open `http://localhost:9222` in a stable/beta/canary Chrome, click the single | 422 * Open `http://localhost:9222` in a stable/beta/canary Chrome, click the single |
425 link to open the devtools with the test loaded. | 423 link to open the devtools with the test loaded. |
426 * You may need to replace devtools.html with inspector.html in your URL (or you | 424 * You may need to replace devtools.html with inspector.html in your URL (or you |
427 can use local chrome inspection of content_shell from chrome://inspect | 425 can use local chrome inspection of content_shell from `chrome://inspect` |
428 instead) | 426 instead) |
429 * In the loaded devtools, set any required breakpoints and execute `test()` in | 427 * In the loaded devtools, set any required breakpoints and execute `test()` in |
430 the console to actually start the test. | 428 the console to actually start the test. |
431 | 429 |
432 ## Rebaselining Layout Tests | 430 ## Rebaselining Layout Tests |
433 | 431 |
434 *** promo | 432 *** promo |
435 To automatically re-baseline tests across all Chromium platforms, using the | 433 To automatically re-baseline tests across all Chromium platforms, using the |
436 buildbot results, see the | 434 buildbot results, see the |
437 [Rebaselining keywords in TestExpectations](./layout_test_expectations.md) | 435 [Rebaselining keywords in TestExpectations](./layout_test_expectations.md) |
(...skipping 36 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
474 See | 472 See |
475 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue
s/list?can=2&q=component%3ABlink%3EInfra) | 473 [bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issue
s/list?can=2&q=component%3ABlink%3EInfra) |
476 for issues related to Blink tools, include the layout test runner. | 474 for issues related to Blink tools, include the layout test runner. |
477 | 475 |
478 * Windows and Linux: Do not copy and paste while the layout tests are running, | 476 * Windows and Linux: Do not copy and paste while the layout tests are running, |
479 as it may interfere with the editing/pasteboard and other clipboard-related | 477 as it may interfere with the editing/pasteboard and other clipboard-related |
480 tests. (Mac tests swizzle NSClipboard to avoid any conflicts). | 478 tests. (Mac tests swizzle NSClipboard to avoid any conflicts). |
481 * If QuickTime is not installed, the plugin tests | 479 * If QuickTime is not installed, the plugin tests |
482 `fast/dom/object-embed-plugin-scripting.html` and | 480 `fast/dom/object-embed-plugin-scripting.html` and |
483 `plugins/embed-attributes-setting.html` are expected to fail. | 481 `plugins/embed-attributes-setting.html` are expected to fail. |
OLD | NEW |