| Index: docs/testing/writing_layout_tests.md
|
| diff --git a/docs/testing/writing_layout_tests.md b/docs/testing/writing_layout_tests.md
|
| index 37e5e2ab7afa861739aaf50c766437d173b43780..6399a7bbe79f94f801a9bec51a2c055c396b619a 100644
|
| --- a/docs/testing/writing_layout_tests.md
|
| +++ b/docs/testing/writing_layout_tests.md
|
| @@ -68,246 +68,24 @@ There are four broad types of layout tests, listed in the order of preference.
|
| Layout tree tests are used as a last resort to test the internal quirks of
|
| the implementation, and they should be avoided in favor of one of the earlier
|
| options.
|
| -## General Principles
|
| -
|
| -The principles below are adapted from
|
| -[Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/docs/test-format-guidelines.html)
|
| -and
|
| -[WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree).
|
| -
|
| -*** note
|
| -This document intentionally uses _should_ a lot more than _must_, as defined in
|
| -[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a
|
| -careful act of balancing many concerns, and this humble document cannot possibly
|
| -capture the context that rests in the head of an experienced Blink engineer.
|
| -***
|
| -
|
| -### Concise
|
| -
|
| -Tests should be **concise**, without compromising on the principles below. Every
|
| -element and piece of code on the page should be necessary and relevant to what
|
| -is being tested. For example, don't build a fully functional signup form if you
|
| -only need a text field or a button.
|
| -
|
| -Content needed to satisfy the principles below is considered necessary. For
|
| -example, it is acceptable and desirable to add elements that make the test
|
| -self-describing (see below), and to add code that makes the test more reliable
|
| -(see below).
|
| -
|
| -Content that makes test failures easier to debug is considered necessary (to
|
| -maintaining a good development speed), and is both acceptable and desirable.
|
| -
|
| -*** promo
|
| -Conciseness is particularly important for reference tests and pixel tests, as
|
| -the test pages are rendered in an 800x600px viewport. Having content outside the
|
| -viewport is undesirable because the outside content does not get compared, and
|
| -because the resulting scrollbars are platform-specific UI widgets, making the
|
| -test results less reliable.
|
| -***
|
| -
|
| -### Fast
|
| -
|
| -Tests should be as **fast** as possible, without compromising on the principles
|
| -below. Blink has several thousand layout tests that are run in parallel, and
|
| -avoiding unnecessary delays is crucial to keeping our Commit Queue in good
|
| -shape.
|
| -
|
| -Avoid
|
| -[window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimers/setTimeout),
|
| -as it wastes time on the testing infrastructure. Instead, use specific event
|
| -handlers, such as
|
| -[window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onload),
|
| -to decide when to advance to the next step in a test.
|
| -
|
| -### Reliable
|
| -
|
| -Tests should be **reliable** and yield consistent results for a given
|
| -implementation. Flaky tests slow down your fellow developers' debugging efforts
|
| -and the Commit Queue.
|
| -
|
| -`window.setTimeout` is again a primary offender here. Asides from wasting time
|
| -on a fast system, tests that rely on fixed timeouts can fail when on systems
|
| -that are slower than expected.
|
| -
|
| -Follow the guidelines in this
|
| -[PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
|
| -
|
| -### Self-Describing
|
| -
|
| -Tests should be **self-describing**, so that a project member can recognize
|
| -whether a test passes or fails without having to read the specification of the
|
| -feature being tested. `testharness.js` makes a test self-describing when used
|
| -correctly, but tests that degrade to manual tests
|
| -[must be carefully designed](http://testthewebforward.org/docs/test-style-guidelines.html)
|
| -to be self-describing.
|
| -
|
| -### Minimal
|
| -
|
| -Tests should require a **minimal** amount of cognitive effort to read and
|
| -maintain.
|
|
|
| -Avoid depending on edge case behavior of features that aren't explicitly covered
|
| -by the test. For example, except where testing parsing, tests should contain
|
| -valid markup (no parsing errors).
|
| -
|
| -Tests should provide as much relevant information as possible when failing.
|
| -`testharness.js` tests should prefer
|
| -[rich assert_ functions](https://github.com/w3c/testharness.js/blob/master/docs/api.md#list-of-assertions)
|
| -to combining `assert_true()` with a boolean operator. Using appropriate
|
| -`assert_` functions results in better diagnostic output when the assertion
|
| -fails.
|
| -
|
| -🚧 Prefer JavaScript's
|
| -[===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comparison_Operators#Identity_strict_equality_())
|
| -operator to
|
| -[==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comparison_Operators#Equality_())
|
| -so that readers don't have to reason about
|
| -[type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abstract-equality-comparison).
|
| -
|
| -*** promo
|
| -The === vs == recommendation is still being discussed on
|
| -[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
|
| -However, please keep in mind that a fellow developer who has to debug a failing
|
| -test may have to consider the
|
| -[special cases for ==](http://dorey.github.io/JavaScript-Equality-Table/) when
|
| -the types of the two values being compared aren't immediately obvious.
|
| -***
|
| -
|
| -### Cross-Platform
|
| -
|
| -Tests should be as **cross-platform** as reasonably possible. Avoid assumptions
|
| -about device type, screen resolution, etc. Unavoidable assumptions should be
|
| -documented.
|
| -
|
| -When possible, tests should only use Web platform features, as specified
|
| -in the relevant standards. When the Web platform's APIs are insufficient,
|
| -tests should prefer to use WPT extended testing APIs, such as
|
| -`wpt_automation`, over Blink-specific testing APIs.
|
| -
|
| -🚧 Tests that use testing APIs should feature-test for the presence of
|
| -those APIs, and gracefully degrade to manual tests (see below) when the testing
|
| -APIs are not available.
|
| +## General Principles
|
|
|
| -Test pages should use the HTML5 doctype (`<!doctype html>`) unless they
|
| -specifically cover
|
| -[quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards_Mode)
|
| -behavior.
|
|
|
| Tests should be written under the assumption that they will be upstreamed
|
| -to the WPT project. For example, tests should follow the
|
| +to the WPT project. To this end, tests should follow the
|
| [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
|
|
|
| -Tests should avoid using features that haven't been shipped by the
|
| -actively-developed major rendering engines (Blink, WebKit, Gecko, Edge). When
|
| -unsure, check [caniuse.com](http://caniuse.com/). By necessity, this
|
| -recommendation does not apply to the feature targeted by the test.
|
| -
|
| -*** note
|
| -It may be tempting have a test for a bleeding-edge feature X depend on feature
|
| -Y, which has only shipped in beta / development versions of various browsers.
|
| -The reasoning would be that all browsers that implement X will have implemented
|
| -Y. Please keep in mind that Chrome has un-shipped features that made it to the
|
| -Beta channel in the past.
|
| -***
|
| -
|
| -Tests that use Blink-specific testing APIs should feature-test for the
|
| -presence of the testing APIs and degrade to
|
| -[manual tests](http://testthewebforward.org/docs/manual-test.html) when
|
| -the testing APIs are not present.
|
| -
|
| -*** promo
|
| -The recommendation to degrade to manual tests is still being discussed on
|
| -[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
|
| -However, please keep in mind that a manual test can be debugged in the browser,
|
| -whereas a test that does not degrade gracefully can only be debugged in the test
|
| -runner. Fellow project members and future you will thank you for having your
|
| -test work as a manual test.
|
| -***
|
| -
|
| -### Self-Contained
|
| -
|
| -Tests must be **self-contained** and not depend on external network resources.
|
| -
|
| -Unless used by multiple test files, CSS and JavaScript should be inlined using
|
| -`<style>` and `<script>` tags. Content shared by multiple tests should be
|
| -placed in a `resources/` directory near the tests that share it. See below for
|
| -using multiple origins in a test.
|
| -
|
| -### File Names
|
|
|
| -Test **file names** should describe what is being tested.
|
| +There is no style guide that applies to all layout tests. However, some projects
|
| +have adopted style guides, such as the
|
| +[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
|
|
|
| -File names should use `snake-case`, but preserve the case of any embedded API
|
| -names. For example, prefer `document-createElement.html` to
|
| -`document-create-element.html`.
|
| -
|
| -### Modern Features
|
| -
|
| -Tests should prefer **modern features** in JavaScript and in the Web Platform,
|
| -provided that they meet the recommendations above for cross-platform tests.
|
| -
|
| -Tests should use
|
| -[strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode)
|
| -for all JavaScript, except when specifically testing sloppy mode behavior.
|
| -Strict mode flags deprecated features and helps catch some errors, such as
|
| -forgetting to declare variables.
|
| -
|
| -🚧 JavaScript code should prefer
|
| -[const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/const)
|
| -and
|
| -[let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/let)
|
| -over `var`,
|
| -[classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Classes)
|
| -over other OOP constructs, and
|
| -[Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
|
| -over other mechanisms for structuring asynchronous code.
|
| -
|
| -*** promo
|
| -The recommendation to prefer `const` and `let` over `var` is currently being
|
| -discussed on
|
| -[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
|
| -***
|
| -
|
| -### Character Encoding
|
| -
|
| -🚧 Tests should use the UTF-8 **character encoding**, which should be
|
| -declared by `<meta charset=utf-8>`. This does not apply when specifically
|
| -testing encodings.
|
| -
|
| -The `<meta>` tag must be the first child of the document's `<head>` element. In
|
| -documents that do not have an explicit `<head>`, the `<meta>` tag must follow
|
| -the doctype.
|
| -
|
| -When HTML pages do not explicitly declare a character encoding, browsers
|
| -determine the encoding using an
|
| -[encoding sniffing algorithm](https://html.spec.whatwg.org/multipage/syntax.html#determining-the-character-encoding)
|
| -that will surprise most modern Web developers. Highlights include a default
|
| -encoding that depends on the user's locale, and non-standardized
|
| -browser-specific heuristics.
|
| -
|
| -*** promo
|
| -The WPT guidelines state that test files that only contain ASCII characters may
|
| -omit the `<meta>` tag. This exception is currently discussed on
|
| -[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
|
| -If taking that route, please keep in mind that Firefox currently issues a
|
| -[development tools](https://developer.mozilla.org/en-US/docs/Tools) warning for
|
| -pages without a declared encoding.
|
| -***
|
| -
|
| -### Coding Style
|
| -
|
| -Tests should aim to have a **coding style** that is consistent with
|
| -[Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.html),
|
| -and
|
| -[Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide.xml),
|
| -with the following exceptions.
|
| -
|
| -* Rules related to Google Closure and JSDoc do not apply.
|
| -* Modern Web Platform and JavaScript features should be preferred to legacy
|
| - constructs that target old browsers.
|
| -* Per the JavaScript guide, new tests should also follow any per-project
|
| - style guide, such as the
|
| - [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
|
| +Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
|
| +important WPT guidelines and highlights some JavaScript concepts that are worth
|
| +paying attention to when trying to infer style rules from existing tests. If
|
| +you're unopinionated and looking for a style guide to follow, the document also
|
| +suggests some defaults.
|
|
|
| ## JavaScript Tests
|
|
|
| @@ -337,10 +115,8 @@ in order to follow the minimality guideline, the test omits the tags `<html>`,
|
|
|
| ```html
|
| <!doctype html>
|
| -<meta charset="utf-8">
|
| -<title>JavaScript: the true literal</title>
|
| +<title>JavaScript: the true literal is immutable and equal to itself</title>
|
| <link rel="help" href="https://tc39.github.io/ecma262/#sec-boolean-literals">
|
| -<meta name="assert" value="The true literal is equal to itself and immutable">
|
| <script src="/resources/testharness.js"></script>
|
| <script src="/resources/testharnessreport.js"></script>
|
| <script>
|
| @@ -375,9 +151,6 @@ promise_test(() => {
|
|
|
| Some points that are not immediately obvious from the example:
|
|
|
| -* The `<meta name="assert">` describes the purpose of the entire file, and
|
| - is not redundant to `<title>`. Don't add a `<meta name="assert">` when the
|
| - information in the `<title>` is sufficient.
|
| * When calling an `assert_` function that compares two values, the first
|
| argument is the actual value (produced by the functionality being tested), and
|
| the second argument is the expected value (known good, golden). The order
|
| @@ -428,6 +201,11 @@ some tests that require a user to perform a gesture, such as a mouse click,
|
| cannot be implemented using Web APIs. The WPT project covers some of these cases
|
| via supplemental testing APIs.
|
|
|
| +When writing tests that rely on supplemental testing APIs, please consider the
|
| +cost and benefits of having the tests
|
| +[gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in
|
| +the absence of the testing APIs.
|
| +
|
| *** promo
|
| In many cases, the user gesture is not actually necessary. For example, many
|
| event handling tests can use
|
| @@ -488,98 +266,6 @@ Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
|
|
|
| [](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
|
|
|
| -### Manual Tests
|
| -
|
| -🚧 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs
|
| -should also be usable as
|
| -[manual tests](http://testthewebforward.org/docs/manual-test.html). This makes
|
| -it easy to debug the test, and to check whether our behavior matches other
|
| -browsers.
|
| -
|
| -*** promo
|
| -The recommendation to degrade to manual tests is still being discussed on
|
| -[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
|
| -However, please keep in mind that a manual test can be debugged in the browser,
|
| -whereas a test that does not degrade gracefully can only be debugged in the test
|
| -runner. Fellow project members and future you will thank you for having your
|
| -test work as a manual test.
|
| -***
|
| -
|
| -Manual tests should minimize the chance of user error. This implies keeping the
|
| -manual steps to a minimum, and having simple and clear instructions that
|
| -describe all the configuration changes and user gestures that match the effect
|
| -of the Blink-specific APIs used by the test.
|
| -
|
| -Below is an example of a fairly minimal test that uses a Blink-Specific API
|
| -(`window.eventSender`), and gracefully degrades to a manual test.
|
| -
|
| -```html
|
| -<!doctype html>
|
| -<meta charset="utf-8">
|
| -<title>DOM: Event.isTrusted for UI events</title>
|
| -<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
|
| -<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
|
| -<meta name="assert"
|
| - content="Event.isTrusted is true for events generated by user interaction">
|
| -<script src="../../resources/testharness.js"></script>
|
| -<script src="../../resources/testharnessreport.js"></script>
|
| -
|
| -<p>Please click on the button below.</p>
|
| -<button>Click Me!</button>
|
| -
|
| -<script>
|
| -'use strict';
|
| -
|
| -setup({ explicit_timeout: true });
|
| -
|
| -promise_test(() => {
|
| - const button = document.querySelector('button');
|
| - return new Promise((resolve, reject) => {
|
| - const button = document.querySelector('button');
|
| - button.addEventListener('click', (event) => {
|
| - resolve(event);
|
| - });
|
| -
|
| - if (window.eventSender) {
|
| - eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
|
| - eventSender.mouseDown();
|
| - eventSender.mouseUp();
|
| - }
|
| - }).then((clickEvent) => {
|
| - assert_true(clickEvent.isTrusted);
|
| - });
|
| -
|
| -}, 'Click generated by user interaction');
|
| -
|
| -</script>
|
| -```
|
| -
|
| -The test exhibits the following desirable features:
|
| -
|
| -* It has a second specification URL (`<link rel="help">`), because the paragraph
|
| - that documents the tested feature (referenced by the primary URL) is not very
|
| - informative on its own.
|
| -* It links to the
|
| - [WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
|
| - rather than to a frozen version of the specification.
|
| -* It contains clear instructions for manually triggering the test conditions.
|
| - The test starts with a paragraph (`<p>`) that tells the tester exactly what to
|
| - do, and the `<button>` that needs to be clicked is clearly labeled.
|
| -* It disables the timeout mechanism built into `testharness.js` by calling
|
| - `setup({ explicit_timeout: true });`
|
| -* It checks for the presence of the Blink-specific testing APIs
|
| - (`window.eventSender`) before invoking them. The test does not automatically
|
| - fail when the APIs are not present.
|
| -* It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
|
| - to separate the test setup from the assertions. This is particularly helpful
|
| - for manual tests that depend on a sequence of events to occur, as Promises
|
| - offer a composable way to express waiting for asynchronous events that avoids
|
| - [callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).
|
| -
|
| -Notice that the test is pretty heavy compared to a minimal JavaScript test that
|
| -does not rely on testing APIs. Only use testing APIs when the desired testing
|
| -conditions cannot be set up using Web Platform APIs.
|
| -
|
| ### Text Test Baselines
|
|
|
| By default, all the test cases in a file that uses `testharness.js` are expected
|
| @@ -704,7 +390,6 @@ The example assumes that the test page is named `ol-reversed.html`.
|
|
|
| ```html
|
| <!doctype html>
|
| -<meta charset="utf-8">
|
| <link rel="match" href="ol-reversed-expected.html">
|
|
|
| <ol reversed>
|
| @@ -718,7 +403,6 @@ The reference page, which must be named `ol-reversed-expected.html`, is below.
|
|
|
| ```html
|
| <!doctype html>
|
| -<meta charset="utf-8">
|
|
|
| <ol>
|
| <li value="3">A</li>
|
| @@ -844,7 +528,6 @@ The following page is an example of a layout tree test.
|
|
|
| ```html
|
| <!doctype html>
|
| -<meta charset="utf-8">
|
| <style>
|
| body { font: 10px Ahem; }
|
| span::after {
|
|
|