New documentation on writing LayoutTests.

docs/testing/layout_test_writing.md

Index: docs/testing/layout_test_writing.md
diff --git a/docs/testing/layout_test_writing.md b/docs/testing/layout_test_writing.md
new file mode 100644
index 0000000000000000000000000000000000000000..901f7681c8f18d40b21e8c694e8b53183bbb45aa
--- /dev/null
+++ b/docs/testing/layout_test_writing.md
@@ -0,0 +1,299 @@
+# Writing Layout Tests
+
+_Layout tests_ is a bit of a misnomer. This term is
+[a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/),
+and we use it to refer to every test that is written as an HTML page and lives
+in [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
+
+Layout tests should be used to accomplish one of the following goals:
+
+1. The entire surface of Blink that is exposed to the Web should be covered by
+   tests that we contribute to the
+   [Web Platform Tests Project](https://github.com/w3c/web-platform-tests)
+   (WPT). This helps us avoid regressions, and gives us confidence that we match
+   other browsers' behavior.
+2. When a Blink feature cannot be tested using the Web Platform, and cannot be
+   easily covered by
+   [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Source/web/tests/?q=webframetest&sq=package:chromium&type=cs),
+   the feature must be covered by layout tests, to avoid unexpected regressions.
+   These tests will use Blink-specific testing APIs that are only available in
+   [content_shell](./layout_tests_in_content_shell.md).
+
+## General Principles
+
+The principles below are adapted from
+[Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/docs/test-format-guidelines.html)
+and
+[WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree).
+
+* Tests should be as **short** as possible. Extraneous elements on the page
+  should be avoided so it is clear what is part of the test.

[spark, 2016/11/10 11:00:11]

"The page should only include elements that are necessary and relevant to what
is being tested." seems slightly better since it's not passive and is more
explicit in describing the reasoning.

[pwnall, 2016/11/10 16:47:14]

Done.

+
+* Tests should be as **fast** as possible. Blink has several thousands layout

[spark, 2016/11/10 11:00:11]

thousand

[pwnall, 2016/11/10 16:47:14]

Done.

+  tests that are run in parallel, and avoiding unnecessary delays is crucial to
+  keeping our Commit Queue in good shape. Avoid [window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimers/setTimeout),
+  as it wastes testing time and can introduce flakiness. Instead, use specific
+  event handlers, such

[spark, 2016/11/10 11:00:11]

such as

[pwnall, 2016/11/10 16:47:14]

Done.

+  [window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onload).
+
+* Tests should be **self-describing**, so that a project member can recognize
+  whether a test passes or fails without having to read the specification of the
+  feature being tested. `testharness.js` makes a test self-describing when used
+  correctly, but tests that degrade to manual tests
+  [must be carefully designed](http://testthewebforward.org/docs/test-style-guidelines.html)
+  to be self-describing.
+
+* Tests should use the **minimal** set of platform features needed to express
+  the test scenario efficiently. Avoid depending on edge case behavior of
+  features that aren't explicitly covered by the test. For example, except where
+  testing parsing, tests should contain no parse errors.
+
+* Tests should be as **cross-platform** as reasonably possible. Avoid
+  assumptions about device type, screen resolution, etc. Unavoidable assumptions
+  should be documented.
+    * When possible, tests should only use Web platform features, as specified
+      in the relevant standards.
+    * Tests should be written under the assumption that they will be upstreamed
+      to the WPT project. For example, tests should follow the
+      [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
+    * Tests that use Blink-specific testing APIs should feature-test for the
+      presence of the testing APIs and degrade to
+      [manual tests](http://testthewebforward.org/docs/manual-test.html)
+      when the testing APIs are not present.
+
+* Tests must be **self-contained** and not depend on external network resources.
+  Unless used by multiple test files, CSS and JavaScript should be inlined using
+  `<style>` and `<script>` tags. Content shared by multiple tests should be
+  placed in a `resources/` directory near the tests that share it. See below for
+  using multiple origins in a test.
+
+* Test **file names** should describe what is being tested.
+
+* Tests must stick to pure ASCII or use the UTF-8 **character encoding**, which
+  should be declared by `<meta charset=utf-8>`. This does not apply when
+  specifically testing encodings.
+
+* Tests must aim to have a **coding style** that is consistent with
+  [Google's JavaScript Style Guide](https://google.github.io/styleguide/javascriptguide.xml),
+  with the following exceptions.
+    * Rules related to Google Closure and JSDoc do not apply.
+    * Modern Web Platform and JavaScript features should be preferred to legacy
+      constructs that target old browsers. For example, prefer `const` and `let`
+      to `var`, and prefer `class` over other OOP constructs.
+    * Concerns regarding buggy behavior in legacy browsers do not apply. For
+      example, the garbage collection cycle note in the _Closures_ section does
+      not apply.
+
+## JavaScript Tests
+
+Whenever possible, the testing criteria should be expressed in JavaScript. The
+alternatives, which will be described in future sections, result in slower and
+less robust tests.
+
+All new JavaScript tests should be written using the
+[testharness.js](https://github.com/w3c/testharness.js/) testing framework. This
+framework is used by the tests in the
+[web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
+which is shared with all the other browser vendors, so `testharness.js` tests
+are more accessible to browser developers.
+
+As a shared framework, `testharness.js` enjoys high-quality documentation, such
+as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and
+[API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.md).
+Layout tets should follow the recommendataions of the above documents.

[spark, 2016/11/10 11:00:11]

"tets" -> "tests", "recommendataions" -> "recommendations"

[pwnall, 2016/11/10 16:47:14]

Done.

+Furthermore, layout tests should include relevant
+[metadata](http://testthewebforward.org/docs/css-metadata.html). The
+specification URL (in `<link rel="help">`) is almost always relevant, and is
+incredibly helpful to a developer who needs to understand the test quickly.
+
+Below is a skeleton for a JavaScript test. Note that, in order to follow the
+minimality guideline, the test omits the tags `<html>`, `<head>` and `<body>`,
+as they can be inferred by the HTML5 parser.
+
+```html
+<!DOCTYPE html>
+<meta charset="utf-8">
+<title>JavaScript core: True is truthy</title>
+<link rel="help"
+      href="http://www.ecma-international.org/ecma-262/6.0/#sec-terms-and-definitions-boolean-value">
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<script>
+test(t => {
+  var x = true;
+  assert_true(x);
+}, 'True is truthy.');
+</script>
+```
+
+### Blink-Specific Testing APIs
+
+Tests that cannot be expressed using the Web Platform APIs rely on
+Blink-specific testing APIs. These APIs are only available in
+[content_shell](./layout_tests_in_content_shell.md).
+
+Whenever possible, tests that rely on Blink-specific testing APIs should also be
+usable as [manual tests](http://testthewebforward.org/docs/manual-test.html).
+This makes it easy to debug the test, and to check whether our behavior matches
+other browsers.
+
+Manual tests should minimize the chance of user error. This implies keeping the
+manual steps to a minimum, and having simple and clear instructions that
+describe all the configuration changes and user gestures which match the effect

[spark, 2016/11/10 11:00:11]

"which" -> "that"

[pwnall, 2016/11/10 16:47:14]

Done.

+of the Blink-specific APIs used by the test.
+
+A downside of Blink-specific APIs is that they are not as well documented as the
+Web Platform features. Learning to use a Blink-specific feature requires finding
+other tests that use it, or reading its source code.
+
+For example, the most popular Blink-specific API is `testRunner`, which is
+implemented in
+[components/test_runner/test_runner.h](../../components/test_runner/test_runner.h)
+and
+[components/test_runner/test_runner.cpp](../../components/test_runner/test_runner.cpp).
+By skimming the `TestRunnerBindings::Install` method, we learn that the
+testRunner API is presented by the `window.testRunner` and
+`window.layoutTestsController` objects, which are synonyms. Reading the
+`TestRunnerBindings::GetObjectTemplateBuilder` method tells us what properties
+are available on the `window.testRunner` object.
+
+*** aside
+`window.testRunner` is the preferred way to access the `testRunner` APIs.
+`window.layoutTestsController` is still supported because it is used by
+3rd-party tests.
+***
+
+See the [components/test_runner/](../../components/test_runner/) directory and
+[WebKit's LayoutTests guide](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree)
+for other useful APIs. For example, `window.eventSender`
+([components/test_runner/event_sender.h](../../components/test_runner/event_sender.h)
+and
+[components/test_runner/event_sender.cpp](../../components/test_runner/event_sender.cpp))
+has methods that simulate events input such as keyboard / mouse input and
+drag-and-drop.
+
+Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
+
+[![UML of testRunner bindings configuring platform implementation](https://docs.google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/svg?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
+
+### Text Test Expectations
+
+By default, all the test cases in a file that uses `testharness.js` are expected
+to pass. However, in some cases, we prefer to add failing test cases to the
+repository, so we can be notified when the failure modes change. In these

[spark, 2016/11/10 11:00:11]

so that

[pwnall, 2016/11/10 16:47:14]

Done.

+situations, a test file will be accompanied by an `-expected.txt` file, which
+documents the test's expected output.
+
+The `-expected.txt` files are generated automatically when appropriate by
+`run-webkit-tests`, which is described [here](./layout_tests.md), and by the
+[rebaselining tools](./layout_test_expectations.md).
+
+`-expected.txt` files should be very rare. In general, layout tests should
+use JavaScript to document Blink's current behavior, rather than using
+JavaScript to document desired behavior and a text file to document current
+behavior.
+
+### js-test tests
+
+*** promo
+For historical reasons, older tests are written using the `js-test` harness.
+This harness is **deprecated**, and should not be used for new tests.
+***
+
+If you need to understand old tests, the best `js-test` documentation is its
+implementation at
+[third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js).
+
+`js-test` tests lean heavily on the Blink-specific `testRunner` testing APIs,
+which will be fully described below. In a nutshell, the tests call
+`testRunner.dumpAsText()` to signal that the page content should be dumped and
+compared against an `-expected.txt` file. As a consequence, `js-test` tests are
+always accompanied by an `-expected.txt`. Asynchronous tests also use
+`testRunner.waitUntilDone()` and `testRunner.notifyDone()` to tell the testing
+tools when they are complete.
+
+### Tests that use a HTTP Server

[spark, 2016/11/10 11:00:11]

an

[pwnall, 2016/11/10 16:47:14]

Done.

+
+By default, tests are loaded as if via `file:` URLs. Some web platform features
+require tests served via HTTP or HTTPS, for example relative paths (`src=/foo`)

[spark, 2016/11/10 11:00:11]

I'd move the examples closer to the thing they exemplify. Perhaps "Some web
platform features, such as relative paths..., require tests served...." Or just
cut off after "HTTP or HTTPS", and start a new sentence, "Examples of such
features include relative paths...."

[pwnall, 2016/11/10 16:47:14]

Ack.
This block of text is extracted from another page. I don't want to change it too
much, because I expect that we'll eventually be describing a different mechanism
for serving HTTP tests (wptserve).

Sorry I forgot to mention this in the review. Thanks for helping make this
better!

+or features restricted to secure protocols.
+
+HTTP tests are those tests that are under `LayoutTests/http/tests` (or virtual
+variants). Use a locally running HTTP server (Apache) to run. Tests are served

[spark, 2016/11/10 11:00:11]

to run them.

[pwnall, 2016/11/10 16:47:14]

Done.

+off of ports 8000, 8080 for HTTP and 8443 for HTTPS. If you run the tests using

[spark, 2016/11/10 11:00:11]

8000 or 8080 for HTTP, and 8443 for HTTPS.

[pwnall, 2016/11/10 16:47:14]

Done.

+`run-webkit-tests`, the server will be started automatically.To run the server

[spark, 2016/11/10 11:00:11]

missing space after "automatically."

[pwnall, 2016/11/10 16:47:14]

Done.

+manually to reproduce or debug a failure:
+
+```bash
+cd src/third_party/WebKit/Tools/Scripts
+run-blink-httpd start
+```
+
+The layout tests will be served from `http://127.0.0.1:8000`. For example, to
+run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
+navigate to
+`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
+tests will behave differently if you go to 127.0.0.1 instead of localhost, so
+use 127.0.0.1.
+
+To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
+or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.
+
+The test server sets up an alias to `LayoutTests/resources` directory. In HTTP
+tests, you can access the testing framework at e.g.
+`src="/js-test-resources/js-test.js"`.
+
+TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
+position to use it to run layout tests.
+
+## Reference Tests
+
+TODO: Check that we match the Web Platform's
+[reftests](http://testthewebforward.org/docs/reftests.html), link to their
+document and summarize it.
+
+## Pixel Tests
+
+`testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and
+`window.testRunner.dumpDragImage()` create an image expectation, turning the
+test into a **pixel test**. These tests have associated `-expected.png` image
+files.
+
+Pixel tests are less robust than JavaScript tests and reference tests, because
+the browser's image output can be influenced by many factors such as the
+graphics driver, the platform's text rendering system, and various
+user-configurable platform settings. For this reason, pixel tests should only be
+used as a last resort.
+
+Pixel tests should still follow the
+[WPT guidelines](http://testthewebforward.org/docs/test-style-guidelines.html).
+The most relevant items are below.
+
+* use a green paragraph / page / square to indicate success
+* use the red color or the word `FAIL` to highlight errors
+* use the [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to
+  minimize the variance introduced by the platform's text rendering system
+
+The following snippet includes the Ahem font in a layout test.
+
+```html
+<style>
+body {
+  font: 10px Ahem;
+}
+</style>
+<script src="/resources/ahem.js"></script>
+```
+
+### Tests that need to paint, raster, or draw a frame of intermediate output
+
+A layout test does not actually draw frames of output until the test exits. If
+it is required to generate a painted frame, then use
+`window.testRunner.displayAsyncThen`, which will run the machinery to put up a
+frame, then call the passed callback. There is also a library at
+`fast/repaint/resources/text-based-repaint.js` to help with writing paint
+invalidation and repaint tests.
+
+## See Also
+
+[Writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit)