Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(395)

Side by Side Diff: doc/configuration.md

Issue 1829483002: Add support for a global configuration file. (Closed) Base URL: git@github.com:dart-lang/test@master
Patch Set: Code review changes Created 4 years, 9 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « CHANGELOG.md ('k') | doc/package_config.md » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 Each package may include a configuration file that applies to the package as a 1 Each package may include a configuration file that applies to the package as a
2 whole. This file can be used to provide custom defaults for various options, to 2 whole. This file can be used to provide custom defaults for various options, to
3 define configuration for multiple files, and more. 3 define configuration for multiple files, and more.
4 4
5 The file is named `dart_test.yaml` and lives at the root of the package, next to 5 The file is named `dart_test.yaml` and lives at the root of the package, next to
6 the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an 6 the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an
7 example: 7 example:
8 8
9 [YAML]: http://yaml.org/ 9 [YAML]: http://yaml.org/
10 10
(...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after
44 * [`reporter`](#reporter) 44 * [`reporter`](#reporter)
45 * [Configuring Tags](#configuring-tags) 45 * [Configuring Tags](#configuring-tags)
46 * [`tags`](#tags) 46 * [`tags`](#tags)
47 * [`add_tags`](#add_tags) 47 * [`add_tags`](#add_tags)
48 * [Configuring Platforms](#configuring-platforms) 48 * [Configuring Platforms](#configuring-platforms)
49 * [`on_os`](#on_os) 49 * [`on_os`](#on_os)
50 * [`on_platform`](#on_platform) 50 * [`on_platform`](#on_platform)
51 * [Configuration Presets](#configuration-presets) 51 * [Configuration Presets](#configuration-presets)
52 * [`presets`](#presets) 52 * [`presets`](#presets)
53 * [`add_preset`](#add_preset) 53 * [`add_preset`](#add_preset)
54 * [Global Configuration](#global-configuration)
54 55
55 ## Test Configuration 56 ## Test Configuration
56 57
57 There are two major categories of configuration field: "test" and "runner". Test 58 There are two major categories of configuration field: "test" and "runner". Test
58 configuration controls how individual tests run, while 59 configuration controls how individual tests run, while
59 [runner configuration](#runner-configuration) controls the test runner as a 60 [runner configuration](#runner-configuration) controls the test runner as a
60 whole. Both types of fields may be used at the top level of a configuration 61 whole. Both types of fields may be used at the top level of a configuration
61 file. However, because different tests can have different test configuration, 62 file. However, because different tests can have different test configuration,
62 only test configuration fields may be used to [configure tags](#tags) or 63 only test configuration fields may be used to [configure tags](#tags) or
63 [platforms](#on_platform). 64 [platforms](#on_platform).
(...skipping 50 matching lines...) Expand 10 before | Expand all | Expand 10 after
114 whether the tests are skipped or a string indicating the reason they're skipped. 115 whether the tests are skipped or a string indicating the reason they're skipped.
115 116
116 [test]: https://www.dartdocs.org/documentation/test/0.12.10+2/test/test.html 117 [test]: https://www.dartdocs.org/documentation/test/0.12.10+2/test/test.html
117 118
118 ```yaml 119 ```yaml
119 tags: 120 tags:
120 chrome: 121 chrome:
121 skip: "Our Chrome launcher is busted. See issue 1234." 122 skip: "Our Chrome launcher is busted. See issue 1234."
122 ``` 123 ```
123 124
125 This field is not supported in the
126 [global configuration file](#global-configuration).
127
124 ### `test_on` 128 ### `test_on`
125 129
126 This field declares which platforms a test supports. It takes a 130 This field declares which platforms a test supports. It takes a
127 [platform selector][platform selectors] and only allows tests to run on 131 [platform selector][platform selectors] and only allows tests to run on
128 platforms that match the selector. It's often used with 132 platforms that match the selector. It's often used with
129 [specific tags](#configuring-tags) to ensure that certain features will only be 133 [specific tags](#configuring-tags) to ensure that certain features will only be
130 tested on supported platforms. 134 tested on supported platforms.
131 135
132 [platform selectors]: https://github.com/dart-lang/test/blob/master/README.md#pl atform-selectors 136 [platform selectors]: https://github.com/dart-lang/test/blob/master/README.md#pl atform-selectors
133 137
134 ```yaml 138 ```yaml
135 tags: 139 tags:
136 # Internet Explorer doesn't support promises yet. 140 # Internet Explorer doesn't support promises yet.
137 promises: {test_on: "browser && !ie"} 141 promises: {test_on: "browser && !ie"}
138 ``` 142 ```
139 143
140 The field can also be used at the top level of the configuration file to 144 The field can also be used at the top level of the configuration file to
141 indicate that the entire package only supports a particular platform. If someone 145 indicate that the entire package only supports a particular platform. If someone
142 tries to run the tests on an unsupported platform, the runner will print a 146 tries to run the tests on an unsupported platform, the runner will print a
143 warning and skip that platform. 147 warning and skip that platform.
144 148
145 ```yaml 149 ```yaml
146 # This package uses dart:io. 150 # This package uses dart:io.
147 test_on: vm 151 test_on: vm
148 ``` 152 ```
149 153
154 This field is not supported in the
155 [global configuration file](#global-configuration).
156
150 ## Runner Configuration 157 ## Runner Configuration
151 158
152 Unlike [test configuration](#test-configuration), runner configuration affects 159 Unlike [test configuration](#test-configuration), runner configuration affects
153 the test runner as a whole rather than individual tests. It can only be used at 160 the test runner as a whole rather than individual tests. It can only be used at
154 the top level of the configuration file. 161 the top level of the configuration file.
155 162
156 ### `paths` 163 ### `paths`
157 164
158 This field indicates the default paths that the test runner should run. These 165 This field indicates the default paths that the test runner should run. These
159 paths are usually directories, although single filenames may be used as well. 166 paths are usually directories, although single filenames may be used as well.
160 Paths must be relative, and they must be in URL format so that they're 167 Paths must be relative, and they must be in URL format so that they're
161 compatible across operating systems. This defaults to `[test]`. 168 compatible across operating systems. This defaults to `[test]`.
162 169
163 ```yaml 170 ```yaml
164 paths: [dart/test] 171 paths: [dart/test]
165 172
166 paths: 173 paths:
167 - test/instantaneous 174 - test/instantaneous
168 - test/fast 175 - test/fast
169 - test/middling 176 - test/middling
170 ``` 177 ```
171 178
179 This field is not supported in the
180 [global configuration file](#global-configuration).
181
172 ### `filename` 182 ### `filename`
173 183
174 This field indicates the filename pattern that the test runner uses to find test 184 This field indicates the filename pattern that the test runner uses to find test
175 files in directories. All files in directories passed on the command line (or in 185 files in directories. All files in directories passed on the command line (or in
176 directories in [`paths`](#paths), if none are passed) whose basenames match this 186 directories in [`paths`](#paths), if none are passed) whose basenames match this
177 pattern will be loaded and run as tests. 187 pattern will be loaded and run as tests.
178 188
179 This supports the full [glob syntax][]. However, since it's only compared 189 This supports the full [glob syntax][]. However, since it's only compared
180 against a path's basename, path separators aren't especially useful. It defaults 190 against a path's basename, path separators aren't especially useful. It defaults
181 to `"*_test.dart"`. 191 to `"*_test.dart"`.
182 192
183 ```yaml 193 ```yaml
184 filename: "test_*.dart" 194 filename: "test_*.dart"
185 ``` 195 ```
186 196
187 [glob syntax]: https://github.com/dart-lang/glob#syntax 197 [glob syntax]: https://github.com/dart-lang/glob#syntax
188 198
199 This field is not supported in the
200 [global configuration file](#global-configuration).
201
189 ### `names` 202 ### `names`
190 203
191 This field causes the runner to only run tests whose names match the given 204 This field causes the runner to only run tests whose names match the given
192 regular expressions. A test's name must match *all* regular expressions in 205 regular expressions. A test's name must match *all* regular expressions in
193 `names`, as well as containing all strings in [`plain_names`](#plain_names), in 206 `names`, as well as containing all strings in [`plain_names`](#plain_names), in
194 order to be run. 207 order to be run.
195 208
196 This is usually used in a [preset](#configuration-presets) to make it possible 209 This is usually used in a [preset](#configuration-presets) to make it possible
197 to quickly select a given set of tests. 210 to quickly select a given set of tests.
198 211
199 ```yaml 212 ```yaml
200 presets: 213 presets:
201 # Pass "-P chrome" to run only Chrome tests. 214 # Pass "-P chrome" to run only Chrome tests.
202 chrome: 215 chrome:
203 names: 216 names:
204 - "^browser:" 217 - "^browser:"
205 - "[Cc]hrome" 218 - "[Cc]hrome"
206 ``` 219 ```
207 220
221 This field is not supported in the
222 [global configuration file](#global-configuration).
223
208 ### `plain_names` 224 ### `plain_names`
209 225
210 This field causes the runner to only run tests whose names contain the given 226 This field causes the runner to only run tests whose names contain the given
211 strings. A test's name must contain *all* strings in `plain_names`, as well as 227 strings. A test's name must contain *all* strings in `plain_names`, as well as
212 matching all regular expressions in [`names`](#names), in order to be run. 228 matching all regular expressions in [`names`](#names), in order to be run.
213 229
214 This is usually used in a [preset](#configuration-presets) to make it possible 230 This is usually used in a [preset](#configuration-presets) to make it possible
215 to quickly select a given set of tests. 231 to quickly select a given set of tests.
216 232
217 ```yaml 233 ```yaml
218 presets: 234 presets:
219 # Pass "-P ie" to run only Internet Explorer tests. 235 # Pass "-P ie" to run only Internet Explorer tests.
220 ie: 236 ie:
221 plain_names: 237 plain_names:
222 - "IE" 238 - "IE"
223 - "Internet Explorer" 239 - "Internet Explorer"
224 ``` 240 ```
225 241
242 This field is not supported in the
243 [global configuration file](#global-configuration).
244
226 ### `include_tags` 245 ### `include_tags`
227 246
228 This field causes the runner to only run tests whose tags match the given 247 This field causes the runner to only run tests whose tags match the given
229 [boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags) 248 [boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags)
230 are used, the exclusions take precedence. 249 are used, the exclusions take precedence.
231 250
232 [boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/RE ADME.md 251 [boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/RE ADME.md
233 252
234 This is usually used in a [preset](#configuration-preset) to make it possible to 253 This is usually used in a [preset](#configuration-preset) to make it possible to
235 quickly select a set of tests. 254 quickly select a set of tests.
236 255
237 ```yaml 256 ```yaml
238 presets: 257 presets:
239 # Pass "-P windowless" to run tests that don't open browser windows. 258 # Pass "-P windowless" to run tests that don't open browser windows.
240 windowless: 259 windowless:
241 include_tags: !browser || content-shell 260 include_tags: !browser || content-shell
242 ``` 261 ```
243 262
263 This field is not supported in the
264 [global configuration file](#global-configuration).
265
244 ### `exclude_tags` 266 ### `exclude_tags`
245 267
246 This field causes the runner not to run tests whose tags match the given 268 This field causes the runner not to run tests whose tags match the given
247 [boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags` 269 [boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags`
248 are used, the exclusions take precedence. 270 are used, the exclusions take precedence.
249 271
250 This is usually used in a [preset](#configuration-preset) to make it possible to 272 This is usually used in a [preset](#configuration-preset) to make it possible to
251 quickly select a set of tests. 273 quickly select a set of tests.
252 274
253 ```yaml 275 ```yaml
254 presets: 276 presets:
255 # Pass "-P windowless" to run tests that don't open browser windows. 277 # Pass "-P windowless" to run tests that don't open browser windows.
256 windowless: 278 windowless:
257 exclude_tags: browser && !content-shell 279 exclude_tags: browser && !content-shell
258 ``` 280 ```
259 281
282 This field is not supported in the
283 [global configuration file](#global-configuration).
284
260 ### `platforms` 285 ### `platforms`
261 286
262 This field indicates which platforms tests should run on by default. It allows 287 This field indicates which platforms tests should run on by default. It allows
263 the same platform identifiers that can be passed to `--platform`. If multiple 288 the same platform identifiers that can be passed to `--platform`. If multiple
264 platforms are included, the test runner will default to running tests on all of 289 platforms are included, the test runner will default to running tests on all of
265 them. This defaults to `[vm]`. 290 them. This defaults to `[vm]`.
266 291
267 ```yaml 292 ```yaml
268 platforms: [content_shell] 293 platforms: [content_shell]
269 294
(...skipping 46 matching lines...) Expand 10 before | Expand all | Expand 10 after
316 ### `reporter` 341 ### `reporter`
317 342
318 This field indicates the default reporter to use. It may be set to "compact", 343 This field indicates the default reporter to use. It may be set to "compact",
319 "expanded", or "json" (although why anyone would want to default to JSON is 344 "expanded", or "json" (although why anyone would want to default to JSON is
320 beyond me). It defaults to "expanded" on Windows and "compact" everywhere else. 345 beyond me). It defaults to "expanded" on Windows and "compact" everywhere else.
321 346
322 ```yaml 347 ```yaml
323 reporter: expanded 348 reporter: expanded
324 ``` 349 ```
325 350
351 This field is not supported in the
352 [global configuration file](#global-configuration).
353
326 ## Configuring Tags 354 ## Configuring Tags
327 355
328 ### `tags` 356 ### `tags`
329 357
330 The `tag` field can be used to apply [test configuration](#test-configuration) 358 The `tag` field can be used to apply [test configuration](#test-configuration)
331 to all tests [with a given tag][tagging tests] or set of tags. It takes a map 359 to all tests [with a given tag][tagging tests] or set of tags. It takes a map
332 from tag selectors to configuration maps. These configuration maps are just like 360 from tag selectors to configuration maps. These configuration maps are just like
333 the top level of the configuration file, except that they may not contain 361 the top level of the configuration file, except that they may not contain
334 [runner configuration](#runner-configuration). 362 [runner configuration](#runner-configuration).
335 363
(...skipping 35 matching lines...) Expand 10 before | Expand all | Expand 10 after
371 Tag configuration is applied at whatever level the tag appears—so if a group is 399 Tag configuration is applied at whatever level the tag appears—so if a group is
372 tagged as `integration`, its timeout will take precedence over the suite's 400 tagged as `integration`, its timeout will take precedence over the suite's
373 timeout but not any tests'. If the group itself had a timeout declared, the 401 timeout but not any tests'. If the group itself had a timeout declared, the
374 group's explicit timeout would take precedence over the tag. 402 group's explicit timeout would take precedence over the tag.
375 403
376 If multiple tags appear at the same level, and they have conflicting 404 If multiple tags appear at the same level, and they have conflicting
377 configurations, the test runner *does not guarantee* what order they'll be 405 configurations, the test runner *does not guarantee* what order they'll be
378 resolved in. In practice, conflicting configuration is pretty unlikely and it's 406 resolved in. In practice, conflicting configuration is pretty unlikely and it's
379 easy to just explicitly specify what you want on the test itself. 407 easy to just explicitly specify what you want on the test itself.
380 408
381 This field counts as [test configuration](#test-configuration). 409 This field counts as [test configuration](#test-configuration). It is not
410 supported in the [global configuration file](#global-configuration).
382 411
383 ### `add_tags` 412 ### `add_tags`
384 413
385 This field adds additional tags. It's technically 414 This field adds additional tags. It's technically
386 [test configuration](#test-configuration), but it's usually used in more 415 [test configuration](#test-configuration), but it's usually used in more
387 specific contexts. For example, when included in a tag's configuration, it can 416 specific contexts. For example, when included in a tag's configuration, it can
388 be used to enable tag inheritance, where adding one tag implicitly adds another 417 be used to enable tag inheritance, where adding one tag implicitly adds another
389 as well. It takes a list of tag name strings. 418 as well. It takes a list of tag name strings.
390 419
391 ```yaml 420 ```yaml
392 tags: 421 tags:
393 # Any test that spawns a browser. 422 # Any test that spawns a browser.
394 browser: 423 browser:
395 timeout: 2x 424 timeout: 2x
396 425
397 # Tests that spawn specific browsers. These automatically get the browser tag 426 # Tests that spawn specific browsers. These automatically get the browser tag
398 # as well. 427 # as well.
399 chrome: {add_tags: [browser]} 428 chrome: {add_tags: [browser]}
400 firefox: {add_tags: [browser]} 429 firefox: {add_tags: [browser]}
401 safari: {add_tags: [browser]} 430 safari: {add_tags: [browser]}
402 ie: {add_tags: [browser]} 431 ie: {add_tags: [browser]}
403 ``` 432 ```
404 433
434 This field is not supported in the
435 [global configuration file](#global-configuration).
436
405 ## Configuring Platforms 437 ## Configuring Platforms
406 438
407 There are two different kinds of platform configuration. 439 There are two different kinds of platform configuration.
408 [Operating system configuration](#on_os) cares about the operating system on 440 [Operating system configuration](#on_os) cares about the operating system on
409 which test runner is running. It sets global configuration for the runner on 441 which test runner is running. It sets global configuration for the runner on
410 particular OSes. [Test platform configuration](#on_platform), on the other hand, 442 particular OSes. [Test platform configuration](#on_platform), on the other hand,
411 cares about the platform the *test* is running on (like the 443 cares about the platform the *test* is running on (like the
412 [`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular 444 [`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular
413 tests that are running on matching platforms. 445 tests that are running on matching platforms.
414 446
(...skipping 107 matching lines...) Expand 10 before | Expand all | Expand 10 after
522 presets: 554 presets:
523 # Shortcut for running only browser tests. 555 # Shortcut for running only browser tests.
524 browser: 556 browser:
525 paths: [test/runner/browser] 557 paths: [test/runner/browser]
526 558
527 # Shortcut for running only Chrome tests. 559 # Shortcut for running only Chrome tests.
528 chrome: 560 chrome:
529 filename: "chrome_*_test.dart" 561 filename: "chrome_*_test.dart"
530 add_presets: [browser] 562 add_presets: [browser]
531 ``` 563 ```
564
565 ## Global Configuration
566
567 The test runner also supports a global configuration file. On Windows, this
568 file's local defaults to `%LOCALAPPDATA%\DartTest.yaml`. On Unix, it defaults to
569 `~/.dart_test.yaml`. It can also be explicitly set using the `DART_TEST_CONFIG`
570 environment variable.
571
572 The global configuration file supports a subset of the fields supported by the
573 package-specific configuration file. In general, it doesn't support fields that
574 are closely tied to the structure of an individual package. Fields that are not
575 supported in the global configuration file say so in their documentation.
OLDNEW
« no previous file with comments | « CHANGELOG.md ('k') | doc/package_config.md » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698