| OLD | NEW |
| 1 Each package may include a configuration file that applies to the package as a | 1 Each package may include a configuration file that applies to the package as a |
| 2 whole. This file can be used to provide custom defaults for various options, to | 2 whole. This file can be used to provide custom defaults for various options, to |
| 3 define configuration for multiple files, and more. | 3 define configuration for multiple files, and more. |
| 4 | 4 |
| 5 The file is named `dart_test.yaml` and lives at the root of the package, next to | 5 The file is named `dart_test.yaml` and lives at the root of the package, next to |
| 6 the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an | 6 the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an |
| 7 example: | 7 example: |
| 8 | 8 |
| 9 [YAML]: http://yaml.org/ | 9 [YAML]: http://yaml.org/ |
| 10 | 10 |
| (...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 44 * [`reporter`](#reporter) | 44 * [`reporter`](#reporter) |
| 45 * [Configuring Tags](#configuring-tags) | 45 * [Configuring Tags](#configuring-tags) |
| 46 * [`tags`](#tags) | 46 * [`tags`](#tags) |
| 47 * [`add_tags`](#add_tags) | 47 * [`add_tags`](#add_tags) |
| 48 * [Configuring Platforms](#configuring-platforms) | 48 * [Configuring Platforms](#configuring-platforms) |
| 49 * [`on_os`](#on_os) | 49 * [`on_os`](#on_os) |
| 50 * [`on_platform`](#on_platform) | 50 * [`on_platform`](#on_platform) |
| 51 * [Configuration Presets](#configuration-presets) | 51 * [Configuration Presets](#configuration-presets) |
| 52 * [`presets`](#presets) | 52 * [`presets`](#presets) |
| 53 * [`add_preset`](#add_preset) | 53 * [`add_preset`](#add_preset) |
| 54 * [Global Configuration](#global-configuration) |
| 54 | 55 |
| 55 ## Test Configuration | 56 ## Test Configuration |
| 56 | 57 |
| 57 There are two major categories of configuration field: "test" and "runner". Test | 58 There are two major categories of configuration field: "test" and "runner". Test |
| 58 configuration controls how individual tests run, while | 59 configuration controls how individual tests run, while |
| 59 [runner configuration](#runner-configuration) controls the test runner as a | 60 [runner configuration](#runner-configuration) controls the test runner as a |
| 60 whole. Both types of fields may be used at the top level of a configuration | 61 whole. Both types of fields may be used at the top level of a configuration |
| 61 file. However, because different tests can have different test configuration, | 62 file. However, because different tests can have different test configuration, |
| 62 only test configuration fields may be used to [configure tags](#tags) or | 63 only test configuration fields may be used to [configure tags](#tags) or |
| 63 [platforms](#on_platform). | 64 [platforms](#on_platform). |
| (...skipping 50 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 114 whether the tests are skipped or a string indicating the reason they're skipped. | 115 whether the tests are skipped or a string indicating the reason they're skipped. |
| 115 | 116 |
| 116 [test]: https://www.dartdocs.org/documentation/test/0.12.10+2/test/test.html | 117 [test]: https://www.dartdocs.org/documentation/test/0.12.10+2/test/test.html |
| 117 | 118 |
| 118 ```yaml | 119 ```yaml |
| 119 tags: | 120 tags: |
| 120 chrome: | 121 chrome: |
| 121 skip: "Our Chrome launcher is busted. See issue 1234." | 122 skip: "Our Chrome launcher is busted. See issue 1234." |
| 122 ``` | 123 ``` |
| 123 | 124 |
| 125 This field is not supported in the |
| 126 [global configuration file](#global-configuration). |
| 127 |
| 124 ### `test_on` | 128 ### `test_on` |
| 125 | 129 |
| 126 This field declares which platforms a test supports. It takes a | 130 This field declares which platforms a test supports. It takes a |
| 127 [platform selector][platform selectors] and only allows tests to run on | 131 [platform selector][platform selectors] and only allows tests to run on |
| 128 platforms that match the selector. It's often used with | 132 platforms that match the selector. It's often used with |
| 129 [specific tags](#configuring-tags) to ensure that certain features will only be | 133 [specific tags](#configuring-tags) to ensure that certain features will only be |
| 130 tested on supported platforms. | 134 tested on supported platforms. |
| 131 | 135 |
| 132 [platform selectors]: https://github.com/dart-lang/test/blob/master/README.md#pl
atform-selectors | 136 [platform selectors]: https://github.com/dart-lang/test/blob/master/README.md#pl
atform-selectors |
| 133 | 137 |
| 134 ```yaml | 138 ```yaml |
| 135 tags: | 139 tags: |
| 136 # Internet Explorer doesn't support promises yet. | 140 # Internet Explorer doesn't support promises yet. |
| 137 promises: {test_on: "browser && !ie"} | 141 promises: {test_on: "browser && !ie"} |
| 138 ``` | 142 ``` |
| 139 | 143 |
| 140 The field can also be used at the top level of the configuration file to | 144 The field can also be used at the top level of the configuration file to |
| 141 indicate that the entire package only supports a particular platform. If someone | 145 indicate that the entire package only supports a particular platform. If someone |
| 142 tries to run the tests on an unsupported platform, the runner will print a | 146 tries to run the tests on an unsupported platform, the runner will print a |
| 143 warning and skip that platform. | 147 warning and skip that platform. |
| 144 | 148 |
| 145 ```yaml | 149 ```yaml |
| 146 # This package uses dart:io. | 150 # This package uses dart:io. |
| 147 test_on: vm | 151 test_on: vm |
| 148 ``` | 152 ``` |
| 149 | 153 |
| 154 This field is not supported in the |
| 155 [global configuration file](#global-configuration). |
| 156 |
| 150 ## Runner Configuration | 157 ## Runner Configuration |
| 151 | 158 |
| 152 Unlike [test configuration](#test-configuration), runner configuration affects | 159 Unlike [test configuration](#test-configuration), runner configuration affects |
| 153 the test runner as a whole rather than individual tests. It can only be used at | 160 the test runner as a whole rather than individual tests. It can only be used at |
| 154 the top level of the configuration file. | 161 the top level of the configuration file. |
| 155 | 162 |
| 156 ### `paths` | 163 ### `paths` |
| 157 | 164 |
| 158 This field indicates the default paths that the test runner should run. These | 165 This field indicates the default paths that the test runner should run. These |
| 159 paths are usually directories, although single filenames may be used as well. | 166 paths are usually directories, although single filenames may be used as well. |
| 160 Paths must be relative, and they must be in URL format so that they're | 167 Paths must be relative, and they must be in URL format so that they're |
| 161 compatible across operating systems. This defaults to `[test]`. | 168 compatible across operating systems. This defaults to `[test]`. |
| 162 | 169 |
| 163 ```yaml | 170 ```yaml |
| 164 paths: [dart/test] | 171 paths: [dart/test] |
| 165 | 172 |
| 166 paths: | 173 paths: |
| 167 - test/instantaneous | 174 - test/instantaneous |
| 168 - test/fast | 175 - test/fast |
| 169 - test/middling | 176 - test/middling |
| 170 ``` | 177 ``` |
| 171 | 178 |
| 179 This field is not supported in the |
| 180 [global configuration file](#global-configuration). |
| 181 |
| 172 ### `filename` | 182 ### `filename` |
| 173 | 183 |
| 174 This field indicates the filename pattern that the test runner uses to find test | 184 This field indicates the filename pattern that the test runner uses to find test |
| 175 files in directories. All files in directories passed on the command line (or in | 185 files in directories. All files in directories passed on the command line (or in |
| 176 directories in [`paths`](#paths), if none are passed) whose basenames match this | 186 directories in [`paths`](#paths), if none are passed) whose basenames match this |
| 177 pattern will be loaded and run as tests. | 187 pattern will be loaded and run as tests. |
| 178 | 188 |
| 179 This supports the full [glob syntax][]. However, since it's only compared | 189 This supports the full [glob syntax][]. However, since it's only compared |
| 180 against a path's basename, path separators aren't especially useful. It defaults | 190 against a path's basename, path separators aren't especially useful. It defaults |
| 181 to `"*_test.dart"`. | 191 to `"*_test.dart"`. |
| 182 | 192 |
| 183 ```yaml | 193 ```yaml |
| 184 filename: "test_*.dart" | 194 filename: "test_*.dart" |
| 185 ``` | 195 ``` |
| 186 | 196 |
| 187 [glob syntax]: https://github.com/dart-lang/glob#syntax | 197 [glob syntax]: https://github.com/dart-lang/glob#syntax |
| 188 | 198 |
| 199 This field is not supported in the |
| 200 [global configuration file](#global-configuration). |
| 201 |
| 189 ### `names` | 202 ### `names` |
| 190 | 203 |
| 191 This field causes the runner to only run tests whose names match the given | 204 This field causes the runner to only run tests whose names match the given |
| 192 regular expressions. A test's name must match *all* regular expressions in | 205 regular expressions. A test's name must match *all* regular expressions in |
| 193 `names`, as well as containing all strings in [`plain_names`](#plain_names), in | 206 `names`, as well as containing all strings in [`plain_names`](#plain_names), in |
| 194 order to be run. | 207 order to be run. |
| 195 | 208 |
| 196 This is usually used in a [preset](#configuration-presets) to make it possible | 209 This is usually used in a [preset](#configuration-presets) to make it possible |
| 197 to quickly select a given set of tests. | 210 to quickly select a given set of tests. |
| 198 | 211 |
| 199 ```yaml | 212 ```yaml |
| 200 presets: | 213 presets: |
| 201 # Pass "-P chrome" to run only Chrome tests. | 214 # Pass "-P chrome" to run only Chrome tests. |
| 202 chrome: | 215 chrome: |
| 203 names: | 216 names: |
| 204 - "^browser:" | 217 - "^browser:" |
| 205 - "[Cc]hrome" | 218 - "[Cc]hrome" |
| 206 ``` | 219 ``` |
| 207 | 220 |
| 221 This field is not supported in the |
| 222 [global configuration file](#global-configuration). |
| 223 |
| 208 ### `plain_names` | 224 ### `plain_names` |
| 209 | 225 |
| 210 This field causes the runner to only run tests whose names contain the given | 226 This field causes the runner to only run tests whose names contain the given |
| 211 strings. A test's name must contain *all* strings in `plain_names`, as well as | 227 strings. A test's name must contain *all* strings in `plain_names`, as well as |
| 212 matching all regular expressions in [`names`](#names), in order to be run. | 228 matching all regular expressions in [`names`](#names), in order to be run. |
| 213 | 229 |
| 214 This is usually used in a [preset](#configuration-presets) to make it possible | 230 This is usually used in a [preset](#configuration-presets) to make it possible |
| 215 to quickly select a given set of tests. | 231 to quickly select a given set of tests. |
| 216 | 232 |
| 217 ```yaml | 233 ```yaml |
| 218 presets: | 234 presets: |
| 219 # Pass "-P ie" to run only Internet Explorer tests. | 235 # Pass "-P ie" to run only Internet Explorer tests. |
| 220 ie: | 236 ie: |
| 221 plain_names: | 237 plain_names: |
| 222 - "IE" | 238 - "IE" |
| 223 - "Internet Explorer" | 239 - "Internet Explorer" |
| 224 ``` | 240 ``` |
| 225 | 241 |
| 242 This field is not supported in the |
| 243 [global configuration file](#global-configuration). |
| 244 |
| 226 ### `include_tags` | 245 ### `include_tags` |
| 227 | 246 |
| 228 This field causes the runner to only run tests whose tags match the given | 247 This field causes the runner to only run tests whose tags match the given |
| 229 [boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags) | 248 [boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags) |
| 230 are used, the exclusions take precedence. | 249 are used, the exclusions take precedence. |
| 231 | 250 |
| 232 [boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/RE
ADME.md | 251 [boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/RE
ADME.md |
| 233 | 252 |
| 234 This is usually used in a [preset](#configuration-preset) to make it possible to | 253 This is usually used in a [preset](#configuration-preset) to make it possible to |
| 235 quickly select a set of tests. | 254 quickly select a set of tests. |
| 236 | 255 |
| 237 ```yaml | 256 ```yaml |
| 238 presets: | 257 presets: |
| 239 # Pass "-P windowless" to run tests that don't open browser windows. | 258 # Pass "-P windowless" to run tests that don't open browser windows. |
| 240 windowless: | 259 windowless: |
| 241 include_tags: !browser || content-shell | 260 include_tags: !browser || content-shell |
| 242 ``` | 261 ``` |
| 243 | 262 |
| 263 This field is not supported in the |
| 264 [global configuration file](#global-configuration). |
| 265 |
| 244 ### `exclude_tags` | 266 ### `exclude_tags` |
| 245 | 267 |
| 246 This field causes the runner not to run tests whose tags match the given | 268 This field causes the runner not to run tests whose tags match the given |
| 247 [boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags` | 269 [boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags` |
| 248 are used, the exclusions take precedence. | 270 are used, the exclusions take precedence. |
| 249 | 271 |
| 250 This is usually used in a [preset](#configuration-preset) to make it possible to | 272 This is usually used in a [preset](#configuration-preset) to make it possible to |
| 251 quickly select a set of tests. | 273 quickly select a set of tests. |
| 252 | 274 |
| 253 ```yaml | 275 ```yaml |
| 254 presets: | 276 presets: |
| 255 # Pass "-P windowless" to run tests that don't open browser windows. | 277 # Pass "-P windowless" to run tests that don't open browser windows. |
| 256 windowless: | 278 windowless: |
| 257 exclude_tags: browser && !content-shell | 279 exclude_tags: browser && !content-shell |
| 258 ``` | 280 ``` |
| 259 | 281 |
| 282 This field is not supported in the |
| 283 [global configuration file](#global-configuration). |
| 284 |
| 260 ### `platforms` | 285 ### `platforms` |
| 261 | 286 |
| 262 This field indicates which platforms tests should run on by default. It allows | 287 This field indicates which platforms tests should run on by default. It allows |
| 263 the same platform identifiers that can be passed to `--platform`. If multiple | 288 the same platform identifiers that can be passed to `--platform`. If multiple |
| 264 platforms are included, the test runner will default to running tests on all of | 289 platforms are included, the test runner will default to running tests on all of |
| 265 them. This defaults to `[vm]`. | 290 them. This defaults to `[vm]`. |
| 266 | 291 |
| 267 ```yaml | 292 ```yaml |
| 268 platforms: [content_shell] | 293 platforms: [content_shell] |
| 269 | 294 |
| (...skipping 46 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 316 ### `reporter` | 341 ### `reporter` |
| 317 | 342 |
| 318 This field indicates the default reporter to use. It may be set to "compact", | 343 This field indicates the default reporter to use. It may be set to "compact", |
| 319 "expanded", or "json" (although why anyone would want to default to JSON is | 344 "expanded", or "json" (although why anyone would want to default to JSON is |
| 320 beyond me). It defaults to "expanded" on Windows and "compact" everywhere else. | 345 beyond me). It defaults to "expanded" on Windows and "compact" everywhere else. |
| 321 | 346 |
| 322 ```yaml | 347 ```yaml |
| 323 reporter: expanded | 348 reporter: expanded |
| 324 ``` | 349 ``` |
| 325 | 350 |
| 351 This field is not supported in the |
| 352 [global configuration file](#global-configuration). |
| 353 |
| 326 ## Configuring Tags | 354 ## Configuring Tags |
| 327 | 355 |
| 328 ### `tags` | 356 ### `tags` |
| 329 | 357 |
| 330 The `tag` field can be used to apply [test configuration](#test-configuration) | 358 The `tag` field can be used to apply [test configuration](#test-configuration) |
| 331 to all tests [with a given tag][tagging tests] or set of tags. It takes a map | 359 to all tests [with a given tag][tagging tests] or set of tags. It takes a map |
| 332 from tag selectors to configuration maps. These configuration maps are just like | 360 from tag selectors to configuration maps. These configuration maps are just like |
| 333 the top level of the configuration file, except that they may not contain | 361 the top level of the configuration file, except that they may not contain |
| 334 [runner configuration](#runner-configuration). | 362 [runner configuration](#runner-configuration). |
| 335 | 363 |
| (...skipping 35 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 371 Tag configuration is applied at whatever level the tag appears—so if a group is | 399 Tag configuration is applied at whatever level the tag appears—so if a group is |
| 372 tagged as `integration`, its timeout will take precedence over the suite's | 400 tagged as `integration`, its timeout will take precedence over the suite's |
| 373 timeout but not any tests'. If the group itself had a timeout declared, the | 401 timeout but not any tests'. If the group itself had a timeout declared, the |
| 374 group's explicit timeout would take precedence over the tag. | 402 group's explicit timeout would take precedence over the tag. |
| 375 | 403 |
| 376 If multiple tags appear at the same level, and they have conflicting | 404 If multiple tags appear at the same level, and they have conflicting |
| 377 configurations, the test runner *does not guarantee* what order they'll be | 405 configurations, the test runner *does not guarantee* what order they'll be |
| 378 resolved in. In practice, conflicting configuration is pretty unlikely and it's | 406 resolved in. In practice, conflicting configuration is pretty unlikely and it's |
| 379 easy to just explicitly specify what you want on the test itself. | 407 easy to just explicitly specify what you want on the test itself. |
| 380 | 408 |
| 381 This field counts as [test configuration](#test-configuration). | 409 This field counts as [test configuration](#test-configuration). It is not |
| 410 supported in the [global configuration file](#global-configuration). |
| 382 | 411 |
| 383 ### `add_tags` | 412 ### `add_tags` |
| 384 | 413 |
| 385 This field adds additional tags. It's technically | 414 This field adds additional tags. It's technically |
| 386 [test configuration](#test-configuration), but it's usually used in more | 415 [test configuration](#test-configuration), but it's usually used in more |
| 387 specific contexts. For example, when included in a tag's configuration, it can | 416 specific contexts. For example, when included in a tag's configuration, it can |
| 388 be used to enable tag inheritance, where adding one tag implicitly adds another | 417 be used to enable tag inheritance, where adding one tag implicitly adds another |
| 389 as well. It takes a list of tag name strings. | 418 as well. It takes a list of tag name strings. |
| 390 | 419 |
| 391 ```yaml | 420 ```yaml |
| 392 tags: | 421 tags: |
| 393 # Any test that spawns a browser. | 422 # Any test that spawns a browser. |
| 394 browser: | 423 browser: |
| 395 timeout: 2x | 424 timeout: 2x |
| 396 | 425 |
| 397 # Tests that spawn specific browsers. These automatically get the browser tag | 426 # Tests that spawn specific browsers. These automatically get the browser tag |
| 398 # as well. | 427 # as well. |
| 399 chrome: {add_tags: [browser]} | 428 chrome: {add_tags: [browser]} |
| 400 firefox: {add_tags: [browser]} | 429 firefox: {add_tags: [browser]} |
| 401 safari: {add_tags: [browser]} | 430 safari: {add_tags: [browser]} |
| 402 ie: {add_tags: [browser]} | 431 ie: {add_tags: [browser]} |
| 403 ``` | 432 ``` |
| 404 | 433 |
| 434 This field is not supported in the |
| 435 [global configuration file](#global-configuration). |
| 436 |
| 405 ## Configuring Platforms | 437 ## Configuring Platforms |
| 406 | 438 |
| 407 There are two different kinds of platform configuration. | 439 There are two different kinds of platform configuration. |
| 408 [Operating system configuration](#on_os) cares about the operating system on | 440 [Operating system configuration](#on_os) cares about the operating system on |
| 409 which test runner is running. It sets global configuration for the runner on | 441 which test runner is running. It sets global configuration for the runner on |
| 410 particular OSes. [Test platform configuration](#on_platform), on the other hand, | 442 particular OSes. [Test platform configuration](#on_platform), on the other hand, |
| 411 cares about the platform the *test* is running on (like the | 443 cares about the platform the *test* is running on (like the |
| 412 [`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular | 444 [`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular |
| 413 tests that are running on matching platforms. | 445 tests that are running on matching platforms. |
| 414 | 446 |
| (...skipping 107 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 522 presets: | 554 presets: |
| 523 # Shortcut for running only browser tests. | 555 # Shortcut for running only browser tests. |
| 524 browser: | 556 browser: |
| 525 paths: [test/runner/browser] | 557 paths: [test/runner/browser] |
| 526 | 558 |
| 527 # Shortcut for running only Chrome tests. | 559 # Shortcut for running only Chrome tests. |
| 528 chrome: | 560 chrome: |
| 529 filename: "chrome_*_test.dart" | 561 filename: "chrome_*_test.dart" |
| 530 add_presets: [browser] | 562 add_presets: [browser] |
| 531 ``` | 563 ``` |
| 564 |
| 565 ## Global Configuration |
| 566 |
| 567 The test runner also supports a global configuration file. On Windows, this |
| 568 file's local defaults to `%LOCALAPPDATA%\DartTest.yaml`. On Unix, it defaults to |
| 569 `~/.dart_test.yaml`. It can also be explicitly set using the `DART_TEST_CONFIG` |
| 570 environment variable. |
| 571 |
| 572 The global configuration file supports a subset of the fields supported by the |
| 573 package-specific configuration file. In general, it doesn't support fields that |
| 574 are closely tied to the structure of an individual package. Fields that are not |
| 575 supported in the global configuration file say so in their documentation. |
| OLD | NEW |