| Index: docs/testing/layout_tests_with_manual_fallback.md
|
| diff --git a/docs/testing/layout_tests_with_manual_fallback.md b/docs/testing/layout_tests_with_manual_fallback.md
|
| new file mode 100644
|
| index 0000000000000000000000000000000000000000..1a51d5a05c0fdd048cbd77bb814da7efa6bf8031
|
| --- /dev/null
|
| +++ b/docs/testing/layout_tests_with_manual_fallback.md
|
| @@ -0,0 +1,119 @@
|
| +# Layout Tests with Manual Fallback
|
| +
|
| +Some Blink features cannot be automatically tested using the Web Platform. Prime
|
| +examples are the APIs that require
|
| +[user activation](https://html.spec.whatwg.org/multipage/interaction.html#triggered-by-user-activation)
|
| +(also known as _a user gesture_), such as [Full Screen](https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API).
|
| +Automated tests for these Blink features must rely on special APIs, which are
|
| +only exposed in testing environments, and are therefore not available in a
|
| +normal browser session.
|
| +
|
| +A popular pattern used in these tests is to rely on the user to perform some
|
| +manual steps in order to run the test case in a normal browser session. These
|
| +tests are effectively
|
| +[manual tests](http://testthewebforward.org/docs/manual-test.html), with
|
| +additional JavaScript code that automatically performs the desired manual steps,
|
| +when loaded in an environment that exposes the needed testing APIs.
|
| +
|
| +## Motivation
|
| +
|
| +Layout tests that degrade to manual tests in the absence of testing APIs have
|
| +the following benefits.
|
| +
|
| +* The manual test component can be debugged in a normal browser session, using
|
| + the rich [developer tools](https://developer.chrome.com/devtools). Tests
|
| + without a manual fallback can only be debugged in the test runner.
|
| +* The manual tests can run in other browsers, making it easy to check whether
|
| + our behavior matches other browsers.
|
| +* The layout tests can form the basis for manual tests that are contributed to
|
| + the [Web Platform Tests Project](https://github.com/w3c/web-platform-tests).
|
| +
|
| +Therefore, the desirability of adding a manual fallback to a test heavily
|
| +depends on whether the feature under test is a Web Platform feature or a
|
| +Blink-only feature, and on the developer's working style. The benefits above
|
| +should be weighed against the added design effort needed to build a manual test,
|
| +and the size and complexity introduced by the manual fallback.
|
| +
|
| +## Development Tips
|
| +
|
| +A natural workflow for writing a layout test that gracefully degrades to a
|
| +manual test is to first develop the manual test in a browser, and then add code
|
| +that feature-checks for testing APIs, and uses them to automate the test's
|
| +manual steps.
|
| +
|
| +Manual tests should minimize the chance of user error. This implies keeping the
|
| +manual steps to a minimum, and having simple and clear instructions that
|
| +describe all the configuration changes and user gestures that match the effect
|
| +of the Blink-specific APIs used by the test.
|
| +
|
| +## Example
|
| +
|
| +Below is an example of a fairly minimal test that uses a Blink-Specific API
|
| +(`window.eventSender`), and gracefully degrades to a manual test.
|
| +
|
| +```html
|
| +<!doctype html>
|
| +<meta charset="utf-8">
|
| +<title>DOM: Event.isTrusted for UI events</title>
|
| +<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
|
| +<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
|
| +<meta name="assert"
|
| + content="Event.isTrusted is true for events generated by user interaction">
|
| +<script src="../../resources/testharness.js"></script>
|
| +<script src="../../resources/testharnessreport.js"></script>
|
| +
|
| +<p>Please click on the button below.</p>
|
| +<button>Click Me!</button>
|
| +
|
| +<script>
|
| +'use strict';
|
| +
|
| +setup({ explicit_timeout: true });
|
| +
|
| +promise_test(() => {
|
| + const button = document.querySelector('button');
|
| + return new Promise((resolve, reject) => {
|
| + const button = document.querySelector('button');
|
| + button.addEventListener('click', (event) => {
|
| + resolve(event);
|
| + });
|
| +
|
| + if (window.eventSender) {
|
| + eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
|
| + eventSender.mouseDown();
|
| + eventSender.mouseUp();
|
| + }
|
| + }).then((clickEvent) => {
|
| + assert_true(clickEvent.isTrusted);
|
| + });
|
| +
|
| +}, 'Click generated by user interaction');
|
| +
|
| +</script>
|
| +```
|
| +
|
| +The test exhibits the following desirable features:
|
| +
|
| +* It has a second specification URL (`<link rel="help">`), because the paragraph
|
| + that documents the tested feature (referenced by the primary URL) is not very
|
| + informative on its own.
|
| +* It links to the
|
| + [WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
|
| + rather than to a frozen version of the specification.
|
| +* It contains clear instructions for manually triggering the test conditions.
|
| + The test starts with a paragraph (`<p>`) that tells the tester exactly what to
|
| + do, and the `<button>` that needs to be clicked is clearly labeled.
|
| +* It disables the timeout mechanism built into `testharness.js` by calling
|
| + `setup({ explicit_timeout: true });`
|
| +* It checks for the presence of the Blink-specific testing APIs
|
| + (`window.eventSender`) before invoking them. The test does not automatically
|
| + fail when the APIs are not present.
|
| +* It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
|
| + to separate the test setup from the assertions. This is particularly helpful
|
| + for manual tests that depend on a sequence of events to occur, as Promises
|
| + offer a composable way to express waiting for asynchronous events that avoids
|
| + [callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).
|
| +
|
| +Notice that the test is pretty heavy compared to a minimal JavaScript test that
|
| +does not rely on testing APIs. Only use testing APIs when the desired testing
|
| +conditions cannot be set up using Web Platform APIs.
|
|
|
Chromium Code Reviews
Issue 2636213003:
Separate layout test docs stating facts from guidelines. (Closed)
