New documentation on writing LayoutTests.

docs/testing/layout_test_writing.md

Index: docs/testing/layout_test_writing.md
diff --git a/docs/testing/layout_test_writing.md b/docs/testing/layout_test_writing.md
new file mode 100644
index 0000000000000000000000000000000000000000..ce541f83764fea7b035fc452ac0f9e2cd2fc6677
--- /dev/null
+++ b/docs/testing/layout_test_writing.md
@@ -0,0 +1,299 @@
+# Writing Layout Tests
+
+_Layout tests_ is a bit of a misnomer. This term is
+[a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/),
+and we use it to refer to every test that is written as an HTML page and lives

[jsbell, 2016/11/10 19:32:26]

We have some ref tests that are SVG documents, as well.

(Also, we may start running imported wpt ones that are e.g. test.worker.js that
gets an owning page for it auto-generated, but I guess that would be covered by
"HTML page" implicitly)

[pwnall, 2016/11/11 20:51:07]

Done.

Added mentions to XHTML and SVG. Let's deal with the .js tests when we get there
:)

+in [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
+
+Layout tests should be used to accomplish one of the following goals:
+
+1. The entire surface of Blink that is exposed to the Web should be covered by
+   tests that we contribute to the
+   [Web Platform Tests Project](https://github.com/w3c/web-platform-tests)
+   (WPT). This helps us avoid regressions, and gives us confidence that we match
+   other browsers' behavior.
+2. When a Blink feature cannot be tested using the Web Platform, and cannot be
+   easily covered by
+   [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Source/web/tests/?q=webframetest&sq=package:chromium&type=cs),
+   the feature must be covered by layout tests, to avoid unexpected regressions.
+   These tests will use Blink-specific testing APIs that are only available in
+   [content_shell](./layout_tests_in_content_shell.md).
+
+## General Principles
+
+The principles below are adapted from
+[Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/docs/test-format-guidelines.html)
+and
+[WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree).
+
+* Tests should be as **short** as possible. The page should only include
+  elements that are necessary and relevant to what is being tested.
+
+* Tests should be as **fast** as possible. Blink has several thousand layout
+  tests that are run in parallel, and avoiding unnecessary delays is crucial to
+  keeping our Commit Queue in good shape. Avoid [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimers/setTimeout),
+  as it wastes testing time and can introduce flakiness. Instead, use specific
+  event handlers, such as
+  [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onload).
+
+* Tests should be **self-describing**, so that a project member can recognize
+  whether a test passes or fails without having to read the specification of the
+  feature being tested. `testharness.js` makes a test self-describing when used
+  correctly, but tests that degrade to manual tests
+  [must be carefully designed](http://testthewebforward.org/docs/test-style-guidelines.html)
+  to be self-describing.
+
+* Tests should use the **minimal** set of platform features needed to express
+  the test scenario efficiently. Avoid depending on edge case behavior of
+  features that aren't explicitly covered by the test. For example, except where
+  testing parsing, tests should contain no parse errors.
+
+* Tests should be as **cross-platform** as reasonably possible. Avoid
+  assumptions about device type, screen resolution, etc. Unavoidable assumptions
+  should be documented.
+    * When possible, tests should only use Web platform features, as specified
+      in the relevant standards.
+    * Tests should be written under the assumption that they will be upstreamed
+      to the WPT project. For example, tests should follow the
+      [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
+    * Tests that use Blink-specific testing APIs should feature-test for the
+      presence of the testing APIs and degrade to
+      [manual tests](http://testthewebforward.org/docs/manual-test.html)
+      when the testing APIs are not present.
+
+* Tests must be **self-contained** and not depend on external network resources.
+  Unless used by multiple test files, CSS and JavaScript should be inlined using
+  `<style>` and `<script>` tags. Content shared by multiple tests should be
+  placed in a `resources/` directory near the tests that share it. See below for
+  using multiple origins in a test.
+
+* Test **file names** should describe what is being tested.
+
+* Tests must stick to pure ASCII or use the UTF-8 **character encoding**, which
+  should be declared by `<meta charset=utf-8>`. This does not apply when
+  specifically testing encodings.
+
+* Tests must aim to have a **coding style** that is consistent with
+  [Google's JavaScript Style Guide](https://google.github.io/styleguide/javascriptguide.xml),
+  with the following exceptions.
+    * Rules related to Google Closure and JSDoc do not apply.
+    * Modern Web Platform and JavaScript features should be preferred to legacy
+      constructs that target old browsers. For example, prefer `const` and `let`
+      to `var`, and prefer `class` over other OOP constructs.
+    * Concerns regarding buggy behavior in legacy browsers do not apply. For
+      example, the garbage collection cycle note in the _Closures_ section does
+      not apply.
+
+## JavaScript Tests
+
+Whenever possible, the testing criteria should be expressed in JavaScript. The
+alternatives, which will be described in future sections, result in slower and
+less robust tests.
+
+All new JavaScript tests should be written using the
+[testharness.js](https://github.com/w3c/testharness.js/) testing framework. This
+framework is used by the tests in the
+[web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
+which is shared with all the other browser vendors, so `testharness.js` tests
+are more accessible to browser developers.
+
+As a shared framework, `testharness.js` enjoys high-quality documentation, such
+as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and
+[API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.md).
+Layout tests should follow the recommendations of the above documents.
+Furthermore, layout tests should include relevant
+[metadata](http://testthewebforward.org/docs/css-metadata.html). The
+specification URL (in `<link rel="help">`) is almost always relevant, and is
+incredibly helpful to a developer who needs to understand the test quickly.
+
+Below is a skeleton for a JavaScript test. Note that, in order to follow the

[jsbell, 2016/11/10 19:32:25]

I think we should give an async_test and promise_test example as well.

[pwnall, 2016/11/11 20:51:07]

Done.

+minimality guideline, the test omits the tags `<html>`, `<head>` and `<body>`,
+as they can be inferred by the HTML5 parser.
+
+```html
+<!DOCTYPE html>
+<meta charset="utf-8">
+<title>JavaScript core: True is truthy</title>
+<link rel="help"
+      href="http://www.ecma-international.org/ecma-262/6.0/#sec-terms-and-definitions-boolean-value">
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<script>
+test(t => {
+  var x = true;
+  assert_true(x);
+}, 'True is truthy.');
+</script>
+```
+
+### Blink-Specific Testing APIs
+
+Tests that cannot be expressed using the Web Platform APIs rely on
+Blink-specific testing APIs. These APIs are only available in
+[content_shell](./layout_tests_in_content_shell.md).
+
+Whenever possible, tests that rely on Blink-specific testing APIs should also be
+usable as [manual tests](http://testthewebforward.org/docs/manual-test.html).
+This makes it easy to debug the test, and to check whether our behavior matches
+other browsers.
+
+Manual tests should minimize the chance of user error. This implies keeping the

[jsbell, 2016/11/10 19:32:26]

We may want an inline example here of a "manual" test if we can come up with a
short one that uses e.g. eventSender.

[pwnall, 2016/11/11 20:51:07]

Done.

+manual steps to a minimum, and having simple and clear instructions that
+describe all the configuration changes and user gestures that match the effect
+of the Blink-specific APIs used by the test.
+
+A downside of Blink-specific APIs is that they are not as well documented as the
+Web Platform features. Learning to use a Blink-specific feature requires finding
+other tests that use it, or reading its source code.
+
+For example, the most popular Blink-specific API is `testRunner`, which is

[jsbell, 2016/11/10 19:32:26]

We maybe want to mention (here or above) that testharness.js-based tests running
in Blink implicitly take advantage of testRunner via the Blink-specific
testharnessreport.js file.

[pwnall, 2016/11/11 20:51:07]

Done.
I added a note.

+implemented in
+[components/test_runner/test_runner.h](../../components/test_runner/test_runner.h)
+and
+[components/test_runner/test_runner.cpp](../../components/test_runner/test_runner.cpp).
+By skimming the `TestRunnerBindings::Install` method, we learn that the
+testRunner API is presented by the `window.testRunner` and
+`window.layoutTestsController` objects, which are synonyms. Reading the
+`TestRunnerBindings::GetObjectTemplateBuilder` method tells us what properties
+are available on the `window.testRunner` object.
+
+*** aside
+`window.testRunner` is the preferred way to access the `testRunner` APIs.
+`window.layoutTestsController` is still supported because it is used by
+3rd-party tests.
+***
+
+See the [components/test_runner/](../../components/test_runner/) directory and
+[WebKit's LayoutTests guide](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree)
+for other useful APIs. For example, `window.eventSender`
+([components/test_runner/event_sender.h](../../components/test_runner/event_sender.h)
+and
+[components/test_runner/event_sender.cpp](../../components/test_runner/event_sender.cpp))
+has methods that simulate events input such as keyboard / mouse input and
+drag-and-drop.
+
+Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
+
+[![UML of testRunner bindings configuring platform implementation](https://docs.google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/svg?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
+
+### Text Test Expectations
+
+By default, all the test cases in a file that uses `testharness.js` are expected
+to pass. However, in some cases, we prefer to add failing test cases to the
+repository, so that we can be notified when the failure modes change. In these

[jsbell, 2016/11/10 19:32:26]

maybe: (e.g. a test starts crashing rather than returning incorrect output)

[pwnall, 2016/11/11 20:51:07]

Done.

+situations, a test file will be accompanied by an `-expected.txt` file, which
+documents the test's expected output.
+
+The `-expected.txt` files are generated automatically when appropriate by
+`run-webkit-tests`, which is described [here](./layout_tests.md), and by the
+[rebaselining tools](./layout_test_expectations.md).
+
+`-expected.txt` files should be very rare. In general, layout tests should
+use JavaScript to document Blink's current behavior, rather than using
+JavaScript to document desired behavior and a text file to document current
+behavior.
+
+### js-test tests
+
+*** promo
+For historical reasons, older tests are written using the `js-test` harness.
+This harness is **deprecated**, and should not be used for new tests.
+***
+
+If you need to understand old tests, the best `js-test` documentation is its
+implementation at
+[third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js).
+
+`js-test` tests lean heavily on the Blink-specific `testRunner` testing APIs,
+which will be fully described below. In a nutshell, the tests call
+`testRunner.dumpAsText()` to signal that the page content should be dumped and
+compared against an `-expected.txt` file. As a consequence, `js-test` tests are
+always accompanied by an `-expected.txt`. Asynchronous tests also use
+`testRunner.waitUntilDone()` and `testRunner.notifyDone()` to tell the testing
+tools when they are complete.
+
+### Tests that use an HTTP Server

[jsbell, 2016/11/10 19:32:25]

Should probably have a section about imported tests here that links to the
relevant docs - those are running w/ wptserve (or will be shortly)

[pwnall, 2016/11/11 20:51:07]

There is a TODO at the end about documenting wptserve. This whole block is
imported as-is from old docs, and I was trying to avoid rewriting it for now.

WDYT? Is it OK to punt rewriting this section, or do I have to address it in
this CL?

[jsbell, 2016/11/14 19:12:22]

Punting is fine, I just missed the TODO. Thanks!

+
+By default, tests are loaded as if via `file:` URLs. Some web platform features
+require tests served via HTTP or HTTPS, for example relative paths (`src=/foo`)

[jsbell, 2016/11/10 19:32:26]

absolute paths

(relative paths work file with file:)

[pwnall, 2016/11/11 20:51:07]

Done.

+or features restricted to secure protocols.
+
+HTTP tests are those tests that are under `LayoutTests/http/tests` (or virtual
+variants). Use a locally running HTTP server (Apache) to run them. Tests are
+served off of ports 8000 and 8080 for HTTP, and 8443 for HTTPS. If you run the
+tests using `run-webkit-tests`, the server will be started automatically. To run
+the server manually to reproduce or debug a failure:
+
+```bash
+cd src/third_party/WebKit/Tools/Scripts
+run-blink-httpd start
+```
+
+The layout tests will be served from `http://127.0.0.1:8000`. For example, to
+run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
+navigate to
+`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
+tests will behave differently if you go to 127.0.0.1 instead of localhost, so
+use 127.0.0.1.
+
+To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
+or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.
+
+The test server sets up an alias to `LayoutTests/resources` directory. In HTTP
+tests, you can access the testing framework at e.g.
+`src="/js-test-resources/js-test.js"`.
+
+TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
+position to use it to run layout tests.
+
+## Reference Tests
+
+TODO: Check that we match the Web Platform's
+[reftests](http://testthewebforward.org/docs/reftests.html), link to their
+document and summarize it.
+
+## Pixel Tests
+
+`testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and
+`window.testRunner.dumpDragImage()` create an image expectation, turning the
+test into a **pixel test**. These tests have associated `-expected.png` image
+files.
+
+Pixel tests are less robust than JavaScript tests and reference tests, because
+the browser's image output can be influenced by many factors such as the
+graphics driver, the platform's text rendering system, and various
+user-configurable platform settings. For this reason, pixel tests should only be
+used as a last resort.
+
+Pixel tests should still follow the
+[WPT guidelines](http://testthewebforward.org/docs/test-style-guidelines.html).
+The most relevant items are below.
+
+* use a green paragraph / page / square to indicate success
+* use the red color or the word `FAIL` to highlight errors
+* use the [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to
+  minimize the variance introduced by the platform's text rendering system
+
+The following snippet includes the Ahem font in a layout test.
+
+```html
+<style>
+body {
+  font: 10px Ahem;
+}
+</style>
+<script src="/resources/ahem.js"></script>
+```
+
+### Tests that need to paint, raster, or draw a frame of intermediate output
+
+A layout test does not actually draw frames of output until the test exits. If
+it is required to generate a painted frame, then use
+`window.testRunner.displayAsyncThen`, which will run the machinery to put up a
+frame, then call the passed callback. There is also a library at
+`fast/repaint/resources/text-based-repaint.js` to help with writing paint
+invalidation and repaint tests.
+
+## See Also
+
+[Writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit)