| OLD | NEW |
|---|
| (Empty) | |
| 1 # Layout Tests with Manual Fallback |
| 2 |
| 3 Some Blink features cannot be automatically tested using the Web Platform. Prime |
| 4 examples are the APIs that require |
| 5 [user activation](https://html.spec.whatwg.org/multipage/interaction.html#trigge
red-by-user-activation) |
| 6 (also known as _a user gesture_), such as [Full Screen](https://developer.mozill
a.org/en-US/docs/Web/API/Fullscreen_API). |
| 7 Automated tests for these Blink features must rely on special APIs, which are |
| 8 only exposed in testing environments, and are therefore not available in a |
| 9 normal browser session. |
| 10 |
| 11 A popular pattern used in these tests is to rely on the user to perform some |
| 12 manual steps in order to run the test case in a normal browser session. These |
| 13 tests are effectively |
| 14 [manual tests](http://testthewebforward.org/docs/manual-test.html), with |
| 15 additional JavaScript code that automatically performs the desired manual steps, |
| 16 when loaded in an environment that exposes the needed testing APIs. |
| 17 |
| 18 ## Motivation |
| 19 |
| 20 Layout tests that degrade to manual tests in the absence of testing APIs have |
| 21 the following benefits. |
| 22 |
| 23 * The manual test component can be debugged in a normal browser session, using |
| 24 the rich [developer tools](https://developer.chrome.com/devtools). Tests |
| 25 without a manual fallback can only be debugged in the test runner. |
| 26 * The manual tests can run in other browsers, making it easy to check whether |
| 27 our behavior matches other browsers. |
| 28 * The layout tests can form the basis for manual tests that are contributed to |
| 29 the [Web Platform Tests Project](https://github.com/w3c/web-platform-tests). |
| 30 |
| 31 Therefore, the desirability of adding a manual fallback to a test heavily |
| 32 depends on whether the feature under test is a Web Platform feature or a |
| 33 Blink-only feature, and on the developer's working style. The benefits above |
| 34 should be weighed against the added design effort needed to build a manual test, |
| 35 and the size and complexity introduced by the manual fallback. |
| 36 |
| 37 ## Development Tips |
| 38 |
| 39 A natural workflow for writing a layout test that gracefully degrades to a |
| 40 manual test is to first develop the manual test in a browser, and then add code |
| 41 that feature-checks for testing APIs, and uses them to automate the test's |
| 42 manual steps. |
| 43 |
| 44 Manual tests should minimize the chance of user error. This implies keeping the |
| 45 manual steps to a minimum, and having simple and clear instructions that |
| 46 describe all the configuration changes and user gestures that match the effect |
| 47 of the Blink-specific APIs used by the test. |
| 48 |
| 49 ## Example |
| 50 |
| 51 Below is an example of a fairly minimal test that uses a Blink-Specific API |
| 52 (`window.eventSender`), and gracefully degrades to a manual test. |
| 53 |
| 54 ```html |
| 55 <!doctype html> |
| 56 <meta charset="utf-8"> |
| 57 <title>DOM: Event.isTrusted for UI events</title> |
| 58 <link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted"> |
| 59 <link rel="help" href="https://dom.spec.whatwg.org/#constructing-events"> |
| 60 <meta name="assert" |
| 61 content="Event.isTrusted is true for events generated by user interaction"> |
| 62 <script src="../../resources/testharness.js"></script> |
| 63 <script src="../../resources/testharnessreport.js"></script> |
| 64 |
| 65 <p>Please click on the button below.</p> |
| 66 <button>Click Me!</button> |
| 67 |
| 68 <script> |
| 69 'use strict'; |
| 70 |
| 71 setup({ explicit_timeout: true }); |
| 72 |
| 73 promise_test(() => { |
| 74 const button = document.querySelector('button'); |
| 75 return new Promise((resolve, reject) => { |
| 76 const button = document.querySelector('button'); |
| 77 button.addEventListener('click', (event) => { |
| 78 resolve(event); |
| 79 }); |
| 80 |
| 81 if (window.eventSender) { |
| 82 eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop); |
| 83 eventSender.mouseDown(); |
| 84 eventSender.mouseUp(); |
| 85 } |
| 86 }).then((clickEvent) => { |
| 87 assert_true(clickEvent.isTrusted); |
| 88 }); |
| 89 |
| 90 }, 'Click generated by user interaction'); |
| 91 |
| 92 </script> |
| 93 ``` |
| 94 |
| 95 The test exhibits the following desirable features: |
| 96 |
| 97 * It has a second specification URL (`<link rel="help">`), because the paragraph |
| 98 that documents the tested feature (referenced by the primary URL) is not very |
| 99 informative on its own. |
| 100 * It links to the |
| 101 [WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_
Standard.22_mean.3F), |
| 102 rather than to a frozen version of the specification. |
| 103 * It contains clear instructions for manually triggering the test conditions. |
| 104 The test starts with a paragraph (`<p>`) that tells the tester exactly what to |
| 105 do, and the `<button>` that needs to be clicked is clearly labeled. |
| 106 * It disables the timeout mechanism built into `testharness.js` by calling |
| 107 `setup({ explicit_timeout: true });` |
| 108 * It checks for the presence of the Blink-specific testing APIs |
| 109 (`window.eventSender`) before invoking them. The test does not automatically |
| 110 fail when the APIs are not present. |
| 111 * It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference
/Global_Objects/Promise) |
| 112 to separate the test setup from the assertions. This is particularly helpful |
| 113 for manual tests that depend on a sequence of events to occur, as Promises |
| 114 offer a composable way to express waiting for asynchronous events that avoids |
| 115 [callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/). |
| 116 |
| 117 Notice that the test is pretty heavy compared to a minimal JavaScript test that |
| 118 does not rely on testing APIs. Only use testing APIs when the desired testing |
| 119 conditions cannot be set up using Web Platform APIs. |
| OLD | NEW |
|---|
Chromium Code Reviews
Issue 2636213003:
Separate layout test docs stating facts from guidelines. (Closed)
