OLD | NEW |
1 Each package may include a configuration file that applies to the package as a | 1 Each package may include a configuration file that applies to the package as a |
2 whole. This file can be used to provide custom defaults for various options, to | 2 whole. This file can be used to provide custom defaults for various options, to |
3 define configuration for multiple files, and more. | 3 define configuration for multiple files, and more. |
4 | 4 |
5 The file is named `dart_test.yaml` and lives at the root of the package, next to | 5 The file is named `dart_test.yaml` and lives at the root of the package, next to |
6 the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an | 6 the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an |
7 example: | 7 example: |
8 | 8 |
9 [YAML]: http://yaml.org/ | 9 [YAML]: http://yaml.org/ |
10 | 10 |
(...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
44 * [`reporter`](#reporter) | 44 * [`reporter`](#reporter) |
45 * [Configuring Tags](#configuring-tags) | 45 * [Configuring Tags](#configuring-tags) |
46 * [`tags`](#tags) | 46 * [`tags`](#tags) |
47 * [`add_tags`](#add_tags) | 47 * [`add_tags`](#add_tags) |
48 * [Configuring Platforms](#configuring-platforms) | 48 * [Configuring Platforms](#configuring-platforms) |
49 * [`on_os`](#on_os) | 49 * [`on_os`](#on_os) |
50 * [`on_platform`](#on_platform) | 50 * [`on_platform`](#on_platform) |
51 * [Configuration Presets](#configuration-presets) | 51 * [Configuration Presets](#configuration-presets) |
52 * [`presets`](#presets) | 52 * [`presets`](#presets) |
53 * [`add_preset`](#add_preset) | 53 * [`add_preset`](#add_preset) |
| 54 * [Global Configuration](#global-configuration) |
54 | 55 |
55 ## Test Configuration | 56 ## Test Configuration |
56 | 57 |
57 There are two major categories of configuration field: "test" and "runner". Test | 58 There are two major categories of configuration field: "test" and "runner". Test |
58 configuration controls how individual tests run, while | 59 configuration controls how individual tests run, while |
59 [runner configuration](#runner-configuration) controls the test runner as a | 60 [runner configuration](#runner-configuration) controls the test runner as a |
60 whole. Both types of fields may be used at the top level of a configuration | 61 whole. Both types of fields may be used at the top level of a configuration |
61 file. However, because different tests can have different test configuration, | 62 file. However, because different tests can have different test configuration, |
62 only test configuration fields may be used to [configure tags](#tags) or | 63 only test configuration fields may be used to [configure tags](#tags) or |
63 [platforms](#on_platform). | 64 [platforms](#on_platform). |
(...skipping 50 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
114 whether the tests are skipped or a string indicating the reason they're skipped. | 115 whether the tests are skipped or a string indicating the reason they're skipped. |
115 | 116 |
116 [test]: https://www.dartdocs.org/documentation/test/0.12.10+2/test/test.html | 117 [test]: https://www.dartdocs.org/documentation/test/0.12.10+2/test/test.html |
117 | 118 |
118 ```yaml | 119 ```yaml |
119 tags: | 120 tags: |
120 chrome: | 121 chrome: |
121 skip: "Our Chrome launcher is busted. See issue 1234." | 122 skip: "Our Chrome launcher is busted. See issue 1234." |
122 ``` | 123 ``` |
123 | 124 |
| 125 This field is not supported in the |
| 126 [global configuration file](#global-configuration). |
| 127 |
124 ### `test_on` | 128 ### `test_on` |
125 | 129 |
126 This field declares which platforms a test supports. It takes a | 130 This field declares which platforms a test supports. It takes a |
127 [platform selector][platform selectors] and only allows tests to run on | 131 [platform selector][platform selectors] and only allows tests to run on |
128 platforms that match the selector. It's often used with | 132 platforms that match the selector. It's often used with |
129 [specific tags](#configuring-tags) to ensure that certain features will only be | 133 [specific tags](#configuring-tags) to ensure that certain features will only be |
130 tested on supported platforms. | 134 tested on supported platforms. |
131 | 135 |
132 [platform selectors]: https://github.com/dart-lang/test/blob/master/README.md#pl
atform-selectors | 136 [platform selectors]: https://github.com/dart-lang/test/blob/master/README.md#pl
atform-selectors |
133 | 137 |
134 ```yaml | 138 ```yaml |
135 tags: | 139 tags: |
136 # Internet Explorer doesn't support promises yet. | 140 # Internet Explorer doesn't support promises yet. |
137 promises: {test_on: "browser && !ie"} | 141 promises: {test_on: "browser && !ie"} |
138 ``` | 142 ``` |
139 | 143 |
140 The field can also be used at the top level of the configuration file to | 144 The field can also be used at the top level of the configuration file to |
141 indicate that the entire package only supports a particular platform. If someone | 145 indicate that the entire package only supports a particular platform. If someone |
142 tries to run the tests on an unsupported platform, the runner will print a | 146 tries to run the tests on an unsupported platform, the runner will print a |
143 warning and skip that platform. | 147 warning and skip that platform. |
144 | 148 |
145 ```yaml | 149 ```yaml |
146 # This package uses dart:io. | 150 # This package uses dart:io. |
147 test_on: vm | 151 test_on: vm |
148 ``` | 152 ``` |
149 | 153 |
| 154 This field is not supported in the |
| 155 [global configuration file](#global-configuration). |
| 156 |
150 ## Runner Configuration | 157 ## Runner Configuration |
151 | 158 |
152 Unlike [test configuration](#test-configuration), runner configuration affects | 159 Unlike [test configuration](#test-configuration), runner configuration affects |
153 the test runner as a whole rather than individual tests. It can only be used at | 160 the test runner as a whole rather than individual tests. It can only be used at |
154 the top level of the configuration file. | 161 the top level of the configuration file. |
155 | 162 |
156 ### `paths` | 163 ### `paths` |
157 | 164 |
158 This field indicates the default paths that the test runner should run. These | 165 This field indicates the default paths that the test runner should run. These |
159 paths are usually directories, although single filenames may be used as well. | 166 paths are usually directories, although single filenames may be used as well. |
160 Paths must be relative, and they must be in URL format so that they're | 167 Paths must be relative, and they must be in URL format so that they're |
161 compatible across operating systems. This defaults to `[test]`. | 168 compatible across operating systems. This defaults to `[test]`. |
162 | 169 |
163 ```yaml | 170 ```yaml |
164 paths: [dart/test] | 171 paths: [dart/test] |
165 | 172 |
166 paths: | 173 paths: |
167 - test/instantaneous | 174 - test/instantaneous |
168 - test/fast | 175 - test/fast |
169 - test/middling | 176 - test/middling |
170 ``` | 177 ``` |
171 | 178 |
| 179 This field is not supported in the |
| 180 [global configuration file](#global-configuration). |
| 181 |
172 ### `filename` | 182 ### `filename` |
173 | 183 |
174 This field indicates the filename pattern that the test runner uses to find test | 184 This field indicates the filename pattern that the test runner uses to find test |
175 files in directories. All files in directories passed on the command line (or in | 185 files in directories. All files in directories passed on the command line (or in |
176 directories in [`paths`](#paths), if none are passed) whose basenames match this | 186 directories in [`paths`](#paths), if none are passed) whose basenames match this |
177 pattern will be loaded and run as tests. | 187 pattern will be loaded and run as tests. |
178 | 188 |
179 This supports the full [glob syntax][]. However, since it's only compared | 189 This supports the full [glob syntax][]. However, since it's only compared |
180 against a path's basename, path separators aren't especially useful. It defaults | 190 against a path's basename, path separators aren't especially useful. It defaults |
181 to `"*_test.dart"`. | 191 to `"*_test.dart"`. |
182 | 192 |
183 ```yaml | 193 ```yaml |
184 filename: "test_*.dart" | 194 filename: "test_*.dart" |
185 ``` | 195 ``` |
186 | 196 |
187 [glob syntax]: https://github.com/dart-lang/glob#syntax | 197 [glob syntax]: https://github.com/dart-lang/glob#syntax |
188 | 198 |
| 199 This field is not supported in the |
| 200 [global configuration file](#global-configuration). |
| 201 |
189 ### `names` | 202 ### `names` |
190 | 203 |
191 This field causes the runner to only run tests whose names match the given | 204 This field causes the runner to only run tests whose names match the given |
192 regular expressions. A test's name must match *all* regular expressions in | 205 regular expressions. A test's name must match *all* regular expressions in |
193 `names`, as well as containing all strings in [`plain_names`](#plain_names), in | 206 `names`, as well as containing all strings in [`plain_names`](#plain_names), in |
194 order to be run. | 207 order to be run. |
195 | 208 |
196 This is usually used in a [preset](#configuration-presets) to make it possible | 209 This is usually used in a [preset](#configuration-presets) to make it possible |
197 to quickly select a given set of tests. | 210 to quickly select a given set of tests. |
198 | 211 |
199 ```yaml | 212 ```yaml |
200 presets: | 213 presets: |
201 # Pass "-P chrome" to run only Chrome tests. | 214 # Pass "-P chrome" to run only Chrome tests. |
202 chrome: | 215 chrome: |
203 names: | 216 names: |
204 - "^browser:" | 217 - "^browser:" |
205 - "[Cc]hrome" | 218 - "[Cc]hrome" |
206 ``` | 219 ``` |
207 | 220 |
| 221 This field is not supported in the |
| 222 [global configuration file](#global-configuration). |
| 223 |
208 ### `plain_names` | 224 ### `plain_names` |
209 | 225 |
210 This field causes the runner to only run tests whose names contain the given | 226 This field causes the runner to only run tests whose names contain the given |
211 strings. A test's name must contain *all* strings in `plain_names`, as well as | 227 strings. A test's name must contain *all* strings in `plain_names`, as well as |
212 matching all regular expressions in [`names`](#names), in order to be run. | 228 matching all regular expressions in [`names`](#names), in order to be run. |
213 | 229 |
214 This is usually used in a [preset](#configuration-presets) to make it possible | 230 This is usually used in a [preset](#configuration-presets) to make it possible |
215 to quickly select a given set of tests. | 231 to quickly select a given set of tests. |
216 | 232 |
217 ```yaml | 233 ```yaml |
218 presets: | 234 presets: |
219 # Pass "-P ie" to run only Internet Explorer tests. | 235 # Pass "-P ie" to run only Internet Explorer tests. |
220 ie: | 236 ie: |
221 plain_names: | 237 plain_names: |
222 - "IE" | 238 - "IE" |
223 - "Internet Explorer" | 239 - "Internet Explorer" |
224 ``` | 240 ``` |
225 | 241 |
| 242 This field is not supported in the |
| 243 [global configuration file](#global-configuration). |
| 244 |
226 ### `include_tags` | 245 ### `include_tags` |
227 | 246 |
228 This field causes the runner to only run tests whose tags match the given | 247 This field causes the runner to only run tests whose tags match the given |
229 [boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags) | 248 [boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags) |
230 are used, the exclusions take precedence. | 249 are used, the exclusions take precedence. |
231 | 250 |
232 [boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/RE
ADME.md | 251 [boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/RE
ADME.md |
233 | 252 |
234 This is usually used in a [preset](#configuration-preset) to make it possible to | 253 This is usually used in a [preset](#configuration-preset) to make it possible to |
235 quickly select a set of tests. | 254 quickly select a set of tests. |
236 | 255 |
237 ```yaml | 256 ```yaml |
238 presets: | 257 presets: |
239 # Pass "-P windowless" to run tests that don't open browser windows. | 258 # Pass "-P windowless" to run tests that don't open browser windows. |
240 windowless: | 259 windowless: |
241 include_tags: !browser || content-shell | 260 include_tags: !browser || content-shell |
242 ``` | 261 ``` |
243 | 262 |
| 263 This field is not supported in the |
| 264 [global configuration file](#global-configuration). |
| 265 |
244 ### `exclude_tags` | 266 ### `exclude_tags` |
245 | 267 |
246 This field causes the runner not to run tests whose tags match the given | 268 This field causes the runner not to run tests whose tags match the given |
247 [boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags` | 269 [boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags` |
248 are used, the exclusions take precedence. | 270 are used, the exclusions take precedence. |
249 | 271 |
250 This is usually used in a [preset](#configuration-preset) to make it possible to | 272 This is usually used in a [preset](#configuration-preset) to make it possible to |
251 quickly select a set of tests. | 273 quickly select a set of tests. |
252 | 274 |
253 ```yaml | 275 ```yaml |
254 presets: | 276 presets: |
255 # Pass "-P windowless" to run tests that don't open browser windows. | 277 # Pass "-P windowless" to run tests that don't open browser windows. |
256 windowless: | 278 windowless: |
257 exclude_tags: browser && !content-shell | 279 exclude_tags: browser && !content-shell |
258 ``` | 280 ``` |
259 | 281 |
| 282 This field is not supported in the |
| 283 [global configuration file](#global-configuration). |
| 284 |
260 ### `platforms` | 285 ### `platforms` |
261 | 286 |
262 This field indicates which platforms tests should run on by default. It allows | 287 This field indicates which platforms tests should run on by default. It allows |
263 the same platform identifiers that can be passed to `--platform`. If multiple | 288 the same platform identifiers that can be passed to `--platform`. If multiple |
264 platforms are included, the test runner will default to running tests on all of | 289 platforms are included, the test runner will default to running tests on all of |
265 them. This defaults to `[vm]`. | 290 them. This defaults to `[vm]`. |
266 | 291 |
267 ```yaml | 292 ```yaml |
268 platforms: [content_shell] | 293 platforms: [content_shell] |
269 | 294 |
(...skipping 46 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
316 ### `reporter` | 341 ### `reporter` |
317 | 342 |
318 This field indicates the default reporter to use. It may be set to "compact", | 343 This field indicates the default reporter to use. It may be set to "compact", |
319 "expanded", or "json" (although why anyone would want to default to JSON is | 344 "expanded", or "json" (although why anyone would want to default to JSON is |
320 beyond me). It defaults to "expanded" on Windows and "compact" everywhere else. | 345 beyond me). It defaults to "expanded" on Windows and "compact" everywhere else. |
321 | 346 |
322 ```yaml | 347 ```yaml |
323 reporter: expanded | 348 reporter: expanded |
324 ``` | 349 ``` |
325 | 350 |
| 351 This field is not supported in the |
| 352 [global configuration file](#global-configuration). |
| 353 |
326 ## Configuring Tags | 354 ## Configuring Tags |
327 | 355 |
328 ### `tags` | 356 ### `tags` |
329 | 357 |
330 The `tag` field can be used to apply [test configuration](#test-configuration) | 358 The `tag` field can be used to apply [test configuration](#test-configuration) |
331 to all tests [with a given tag][tagging tests] or set of tags. It takes a map | 359 to all tests [with a given tag][tagging tests] or set of tags. It takes a map |
332 from tag selectors to configuration maps. These configuration maps are just like | 360 from tag selectors to configuration maps. These configuration maps are just like |
333 the top level of the configuration file, except that they may not contain | 361 the top level of the configuration file, except that they may not contain |
334 [runner configuration](#runner-configuration). | 362 [runner configuration](#runner-configuration). |
335 | 363 |
(...skipping 35 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
371 Tag configuration is applied at whatever level the tag appears—so if a group is | 399 Tag configuration is applied at whatever level the tag appears—so if a group is |
372 tagged as `integration`, its timeout will take precedence over the suite's | 400 tagged as `integration`, its timeout will take precedence over the suite's |
373 timeout but not any tests'. If the group itself had a timeout declared, the | 401 timeout but not any tests'. If the group itself had a timeout declared, the |
374 group's explicit timeout would take precedence over the tag. | 402 group's explicit timeout would take precedence over the tag. |
375 | 403 |
376 If multiple tags appear at the same level, and they have conflicting | 404 If multiple tags appear at the same level, and they have conflicting |
377 configurations, the test runner *does not guarantee* what order they'll be | 405 configurations, the test runner *does not guarantee* what order they'll be |
378 resolved in. In practice, conflicting configuration is pretty unlikely and it's | 406 resolved in. In practice, conflicting configuration is pretty unlikely and it's |
379 easy to just explicitly specify what you want on the test itself. | 407 easy to just explicitly specify what you want on the test itself. |
380 | 408 |
381 This field counts as [test configuration](#test-configuration). | 409 This field counts as [test configuration](#test-configuration). It is not |
| 410 supported in the [global configuration file](#global-configuration). |
382 | 411 |
383 ### `add_tags` | 412 ### `add_tags` |
384 | 413 |
385 This field adds additional tags. It's technically | 414 This field adds additional tags. It's technically |
386 [test configuration](#test-configuration), but it's usually used in more | 415 [test configuration](#test-configuration), but it's usually used in more |
387 specific contexts. For example, when included in a tag's configuration, it can | 416 specific contexts. For example, when included in a tag's configuration, it can |
388 be used to enable tag inheritance, where adding one tag implicitly adds another | 417 be used to enable tag inheritance, where adding one tag implicitly adds another |
389 as well. It takes a list of tag name strings. | 418 as well. It takes a list of tag name strings. |
390 | 419 |
391 ```yaml | 420 ```yaml |
392 tags: | 421 tags: |
393 # Any test that spawns a browser. | 422 # Any test that spawns a browser. |
394 browser: | 423 browser: |
395 timeout: 2x | 424 timeout: 2x |
396 | 425 |
397 # Tests that spawn specific browsers. These automatically get the browser tag | 426 # Tests that spawn specific browsers. These automatically get the browser tag |
398 # as well. | 427 # as well. |
399 chrome: {add_tags: [browser]} | 428 chrome: {add_tags: [browser]} |
400 firefox: {add_tags: [browser]} | 429 firefox: {add_tags: [browser]} |
401 safari: {add_tags: [browser]} | 430 safari: {add_tags: [browser]} |
402 ie: {add_tags: [browser]} | 431 ie: {add_tags: [browser]} |
403 ``` | 432 ``` |
404 | 433 |
| 434 This field is not supported in the |
| 435 [global configuration file](#global-configuration). |
| 436 |
405 ## Configuring Platforms | 437 ## Configuring Platforms |
406 | 438 |
407 There are two different kinds of platform configuration. | 439 There are two different kinds of platform configuration. |
408 [Operating system configuration](#on_os) cares about the operating system on | 440 [Operating system configuration](#on_os) cares about the operating system on |
409 which test runner is running. It sets global configuration for the runner on | 441 which test runner is running. It sets global configuration for the runner on |
410 particular OSes. [Test platform configuration](#on_platform), on the other hand, | 442 particular OSes. [Test platform configuration](#on_platform), on the other hand, |
411 cares about the platform the *test* is running on (like the | 443 cares about the platform the *test* is running on (like the |
412 [`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular | 444 [`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular |
413 tests that are running on matching platforms. | 445 tests that are running on matching platforms. |
414 | 446 |
(...skipping 107 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
522 presets: | 554 presets: |
523 # Shortcut for running only browser tests. | 555 # Shortcut for running only browser tests. |
524 browser: | 556 browser: |
525 paths: [test/runner/browser] | 557 paths: [test/runner/browser] |
526 | 558 |
527 # Shortcut for running only Chrome tests. | 559 # Shortcut for running only Chrome tests. |
528 chrome: | 560 chrome: |
529 filename: "chrome_*_test.dart" | 561 filename: "chrome_*_test.dart" |
530 add_presets: [browser] | 562 add_presets: [browser] |
531 ``` | 563 ``` |
| 564 |
| 565 ## Global Configuration |
| 566 |
| 567 The test runner also supports a global configuration file. On Windows, this |
| 568 file's local defaults to `%LOCALAPPDATA%\DartTest.yaml`. On Unix, it defaults to |
| 569 `~/.dart_test.yaml`. It can also be explicitly set using the `DART_TEST_CONFIG` |
| 570 environment variable. |
| 571 |
| 572 The global configuration file supports a subset of the fields supported by the |
| 573 package-specific configuration file. In general, it doesn't support fields that |
| 574 are closely tied to the structure of an individual package. Fields that are not |
| 575 supported in the global configuration file say so in their documentation. |
OLD | NEW |