OLD | NEW |
(Empty) | |
| 1 <!-- |
| 2 Copyright (c) 2016, the Dart project authors. Please see the AUTHORS file |
| 3 for details. All rights reserved. Use of this source code is governed by a |
| 4 BSD-style license that can be found in the LICENSE file. |
| 5 --> |
| 6 # Test Infrastructure without Batteries |
| 7 |
| 8 This package: |
| 9 |
| 10 * Provides a way to test a compiler in multiple steps. |
| 11 |
| 12 * Provides a way to run standalone tests. A standalone test is a test that has
a `main` method, and can be run as a standalone program. |
| 13 |
| 14 * Ensures all tests and implementations are free of warnings (using dartanalyz
er). |
| 15 |
| 16 This package is not: |
| 17 |
| 18 * A replacement for `package:test`. This package can be used to run `package:t
est` tests, and `package:test` can be viewed as the batteries that aren't includ
ed in this package. |
| 19 |
| 20 ## Motivation |
| 21 |
| 22 We want to test tool chains, for example, a Dart compiler. Depending on the tool
chain, it may comprise several individual steps. For example, to test dart2js,
you have these steps: |
| 23 |
| 24 1. Run dart2js on a Dart source file to produce a Javascript output file. |
| 25 |
| 26 2. Run the Javascript file from step 1 on a Javascript interpreter and report
if the program threw an exception. |
| 27 |
| 28 On the other hand, to test a Dart VM, there's only one step: |
| 29 |
| 30 1. Run the Dart source file in the Dart VM and report if the program threw an
exception. |
| 31 |
| 32 Similarly, to test dartanalyzer, there's also a single step: |
| 33 |
| 34 1. Analyze the Dart source file and report if there were any problems. |
| 35 |
| 36 In general, a tool chain can have more steps, for example, a pub transformer. |
| 37 |
| 38 Furthermore, multiple tool chains may share the input sources and should agree o
n the behavior. For example, you should be able to compile `hello-world.dart` wi
th dart2js and run it on d8 and it shouldn't throw an exception, running `hello-
world.dart` on the Dart VM shouldn't throw an exception, and analysing it with d
artanalyzer should report nothing. |
| 39 |
| 40 In addition, parts of the tool chain may have been implemented in Dart and have
unit tests written in Dart, for example, using [package:test](https://github.com
/dart-lang/test). We want to run these unit tests, and have noticed that compile
r unit tests in general run faster when run from the same Dart VM process (due t
o dynamic optimizations kicking in). For this reason, it's convenient to have a
single Dart program that runs all tests. On the other hand, when developing, it'
s often convenient to run just a single test. |
| 41 |
| 42 For this reason, we want to support running unit tests individually, or combined
in one program. And we also want the Dart-based implementation to be free of pr
oblems with respect to dartanalyzer. |
| 43 |
| 44 ## Test Suites |
| 45 |
| 46 A test suite is a collection of tests. Based on the above motivation, we have tw
o kinds of suites: |
| 47 |
| 48 1. [Chain](#Chain), a test suite for tool chains. |
| 49 |
| 50 2. [Dart](#Dart), a test suite for Dart-based unit tests. |
| 51 |
| 52 ## Getting Started |
| 53 |
| 54 1. Create a [configuration file](#Configuration) named `testing.json`. |
| 55 |
| 56 2. Run `bin/run_tests.dart`. |
| 57 |
| 58 ## Configuration |
| 59 |
| 60 The test runner is configured using a JSON file. A minimal configuration file is
: |
| 61 |
| 62 ```json |
| 63 { |
| 64 } |
| 65 ``` |
| 66 |
| 67 ### Chain |
| 68 |
| 69 A `Chain` suite is a suite that's designed to test a tool chain and can be used
to test anything that can be divided into one or more steps. |
| 70 |
| 71 Here a complete example of a `Chain` suite: |
| 72 |
| 73 ```json |
| 74 { |
| 75 "suites": [ |
| 76 { |
| 77 "name": "golden", |
| 78 "kind": "Chain", |
| 79 "source": "test/golden_suite.dart", |
| 80 "path": "test/golden/", |
| 81 "status": "test/golden.status", |
| 82 "pattern": [ |
| 83 "\\.dart$" |
| 84 ], |
| 85 "exclude": [ |
| 86 ] |
| 87 } |
| 88 ] |
| 89 } |
| 90 ``` |
| 91 |
| 92 The properties of a `Chain` suite are: |
| 93 |
| 94 *name*: a name for the suite. For simple packages, `test` or the package name ar
e good candidates. In the Dart SDK, for example, it would be things like `langua
ge`, `corelib`, etc. |
| 95 |
| 96 *kind*: always `Chain` for this kind of suite. |
| 97 |
| 98 *source*: a relative URI to a Dart program that implements the steps in the suit
e. See [below](#Implementing-a-Chain-Suite). |
| 99 |
| 100 *path*: a URI relative to the configuration file which is the root directory of
all files in this suite. For now, only file URIs are supported. Each file is pas
sed to the first step in the suite. |
| 101 |
| 102 *status*: a URI relative to the configuration file which lists the status of tes
ts. |
| 103 |
| 104 *pattern*: a list of regular expressions that match file names that are tests. |
| 105 |
| 106 *exclude*: a list of regular expressions that exclude files from being included
in this suite. |
| 107 |
| 108 #### Implementing a Chain Suite |
| 109 |
| 110 The `source` property of a `Chain` suite is a Dart program that must provide a t
op-level method with this name and signature: |
| 111 |
| 112 ```dart |
| 113 Future<ChainContext> createContext(Chain suite) async { ... } |
| 114 ``` |
| 115 |
| 116 A suite is expected to implement a subclass of `ChainContext` which defines the
steps that make up the chain and return it from `createContext`. |
| 117 |
| 118 A step is a subclass of `Step`. The input to the first step is a `TestDescriptio
n`. The input to step n+1 is the output of step n. |
| 119 |
| 120 Here is an example of a suite that runs tests on the Dart VM: |
| 121 |
| 122 ```dart |
| 123 import 'testing.dart'; |
| 124 |
| 125 Future<ChainContext> createContext( |
| 126 Chain suite, Map<String, String> enviroment) async { |
| 127 return new VmContext(); |
| 128 } |
| 129 |
| 130 class VmContext extends ChainContext { |
| 131 final List<Step> steps = const <Step>[const DartVmStep()]; |
| 132 } |
| 133 |
| 134 class DartVmStep extends Step<TestDescription, int, VmContext> { |
| 135 const DartVmStep(); |
| 136 |
| 137 String get name => "Dart VM"; |
| 138 |
| 139 Future<Result<int>> run(TestDescription input, VmContext context) async { |
| 140 StdioProcess process = await StdioProcess.run("dart", [input.file.path]); |
| 141 return process.toResult(); |
| 142 } |
| 143 } |
| 144 |
| 145 main(List<String> arguments) => runMe(arguments, createContext); |
| 146 ``` |
| 147 |
| 148 An example with multiple steps in the chain can be found in the Kernel package's
[suite](https://github.com/dart-lang/kernel/blob/closure_conversion/test/closur
es/suite.dart). Notice how this suite stores an `AnalysisContext` in its `TestCo
ntext` and is this way able to reuse the same `AnalysisContext` in all tests. |
| 149 |
| 150 ### Dart |
| 151 |
| 152 The `Dart` suite is for running unit tests written in Dart. Each test is a Dart
program with a main method that can be run directly from the command line. |
| 153 |
| 154 The suite generates a new Dart program which combines all the tests included in
the suite, so they can all be run (in sequence) in the same process. Such tests
must be co-operative and must clean up after themselves. |
| 155 |
| 156 You can use any test-framework, for example, `package:test` in these individual
programs, as long as the frameworks are well-behaved with respect to global stat
e, see [below](#Well-Behaved-Tests). |
| 157 |
| 158 Here is a complete example of a `Dart` suite: |
| 159 |
| 160 ```json |
| 161 { |
| 162 "suites": [ |
| 163 { |
| 164 "name": "my-package", |
| 165 "path": "test/", |
| 166 "kind": "Dart", |
| 167 "pattern": [ |
| 168 "_test\\.dart$" |
| 169 ], |
| 170 "exclude": [ |
| 171 "/test/golden/" |
| 172 ] |
| 173 } |
| 174 ] |
| 175 } |
| 176 ``` |
| 177 |
| 178 The properties of a `Dart` suite are: |
| 179 |
| 180 *name*: a name for the suite. For simple packages, `test` or the package name ar
e good candidates. In the Dart SDK, for example, the names could be the name of
the component that's tested by this suite's unit tests, for example, `dart2js`. |
| 181 |
| 182 *path*: a URI relative to the configuration file which is the root directory of
all files in this suite. For now, only file URIs are supported. |
| 183 |
| 184 *kind*: always `Dart` for this kind of suite. |
| 185 |
| 186 *pattern*: a list of regular expressions that match file names that are tests. |
| 187 |
| 188 *exclude*: a list of regular expressions that exclude files from being included
in this suite. |
| 189 |
| 190 #### Well-Behaved Tests |
| 191 |
| 192 The `Dart` suite makes certain assumptions about the tests it runs. |
| 193 |
| 194 * All tests use the same packages configuration file. |
| 195 |
| 196 * An asynchronous test returns a `Future` from its `main`. |
| 197 |
| 198 * Tests manages global state. |
| 199 |
| 200 All tests are imported into the same program as individual libraries, which is w
hy they all must use the same `.packages` file. The tests aren't concatenated, s
o they have the lexical scope you'd normally expect from a Dart library. |
| 201 |
| 202 Tests are run in order. In particular, the test framework will not start the nex
t test until any future returned from the current test's `main` method complete.
In addition, asynchronous tests are expected to have finished all asynchronous
operations when the future returned from their `main` method completes (with or
without an error). |
| 203 |
| 204 Tests are expected to manage global state (aka static state). Simply put: clean
up after yourself. But if it's simpler to ensure global state is reset before ru
nning a test and not clean up afterwards, that's also fine as long as all tests
agree on how to manage their shared global state. |
| 205 |
| 206 ### Configuring Analyzed Programs |
| 207 |
| 208 By default, all tests in `Dart` suites are analyzed by the `dartanalyzer`. It is
possible to exclude tests from analysis, and it's possible to add additional fi
les to be analyzed. Here is a complete example of a `Dart` suite and analyzer co
nfiguration: |
| 209 |
| 210 ```json |
| 211 { |
| 212 "suites": [ |
| 213 { |
| 214 "name": "my-package", |
| 215 "path": "test/", |
| 216 "kind": "Dart", |
| 217 "pattern": [ |
| 218 "_test\\.dart$" |
| 219 ], |
| 220 "exclude": [ |
| 221 "/test/golden/" |
| 222 ] |
| 223 } |
| 224 ], |
| 225 "analyze": { |
| 226 "uris": [ |
| 227 "lib/", |
| 228 ], |
| 229 "exclude": [ |
| 230 "/third_party/" |
| 231 ] |
| 232 } |
| 233 } |
| 234 ``` |
| 235 |
| 236 The properties of the `analyze` section are: |
| 237 |
| 238 *uris*: a list of URIs relative to the configuration file that should also be an
alyzed. For now, only file URIs are supported. |
| 239 |
| 240 *exclude*: a list of regular expression that matches file names that should be e
xcluded from analysis. For now, the files are still analyzed but diagnostics are
suppressed and ignored. |
| 241 |
| 242 ## Integrating with Other Test Runners |
| 243 |
| 244 ### `test.dart` |
| 245 |
| 246 To run the suite `my_suite` from `test.dart`, create a file named `mysuite_test.
dart` with this content: |
| 247 |
| 248 import 'package:async_helper/async_helper.dart' show asyncTest; |
| 249 |
| 250 import 'package:testing/testing.dart' show run; |
| 251 |
| 252 main(List<String> arguments) => asyncTest(run(arguments, ["my_suite"])); |
| 253 |
| 254 ### `package:test` |
| 255 |
| 256 To run the suite `my_suite` from `package:test`, create a file named `mysuite_te
st.dart` with this content: |
| 257 |
| 258 import 'package:test/test.dart' show Timeout, test; |
| 259 |
| 260 import 'package:testing/testing.dart' show run; |
| 261 |
| 262 main() { |
| 263 test("my_suite", () => run([], ["my_suite"]), |
| 264 timeout: new Timeout(new Duration(minutes: 5))); |
| 265 } |
OLD | NEW |