| Index: doc/package_config.md
|
| diff --git a/doc/package_config.md b/doc/package_config.md
|
| deleted file mode 100644
|
| index 974e7ce98269bd8a3ed418d125ae44fb60b52160..0000000000000000000000000000000000000000
|
| --- a/doc/package_config.md
|
| +++ /dev/null
|
| @@ -1,531 +0,0 @@
|
| -Each package may include a configuration file that applies to the package as a
|
| -whole. This file can be used to provide custom defaults for various options, to
|
| -define configuration for multiple files, and more.
|
| -
|
| -The file is named `dart_test.yaml` and lives at the root of the package, next to
|
| -the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an
|
| -example:
|
| -
|
| -[YAML]: http://yaml.org/
|
| -
|
| -```yaml
|
| -# This package's tests are very slow. Double the default timeout.
|
| -timeout: 2x
|
| -
|
| -# This is a browser-only package, so test on content shell by default.
|
| -platforms: [content-shell]
|
| -
|
| -tags:
|
| - # Integration tests are even slower, so increase the timeout again.
|
| - integration: {timeout: 2x}
|
| -
|
| - # Sanity tests are quick and verify that nothing is obviously wrong. Declaring
|
| - # the tag allows us to tag tests with it without getting warnings.
|
| - sanity:
|
| -```
|
| -
|
| -* [Test Configuration](#test-configuration)
|
| - * [`timeout`](#timeout)
|
| - * [`verbose_trace`](#verbose_trace)
|
| - * [`js_trace`](#js_trace)
|
| - * [`skip`](#skip)
|
| - * [`test_on`](#test_on)
|
| -* [Runner Configuration](#runner-configuration)
|
| - * [`paths`](#paths)
|
| - * [`filename`](#filename)
|
| - * [`names`](#names)
|
| - * [`plain_names`](#plain_names)
|
| - * [`include_tags`](#include_tags)
|
| - * [`exclude_tags`](#exclude_tags)
|
| - * [`platforms`](#platforms)
|
| - * [`concurrency`](#concurrency)
|
| - * [`pause_after_load`](#pause_after_load)
|
| - * [`pub_serve`](#pub_serve)
|
| - * [`reporter`](#reporter)
|
| -* [Configuring Tags](#configuring-tags)
|
| - * [`tags`](#tags)
|
| - * [`add_tags`](#add_tags)
|
| -* [Configuring Platforms](#configuring-platforms)
|
| - * [`on_os`](#on_os)
|
| - * [`on_platform`](#on_platform)
|
| -* [Configuration Presets](#configuration-presets)
|
| - * [`presets`](#presets)
|
| - * [`add_preset`](#add_preset)
|
| -
|
| -## Test Configuration
|
| -
|
| -There are two major categories of configuration field: "test" and "runner". Test
|
| -configuration controls how individual tests run, while
|
| -[runner configuration](#runner-configuration) controls the test runner as a
|
| -whole. Both types of fields may be used at the top level of a configuration
|
| -file. However, because different tests can have different test configuration,
|
| -only test configuration fields may be used to [configure tags](#tags) or
|
| -[platforms](#on_platform).
|
| -
|
| -### `timeout`
|
| -
|
| -This field indicates how much time the test runner should allow a test to remain
|
| -inactive before it considers that test to have failed. It has three possible
|
| -formats:
|
| -
|
| -* The string "none" indicates that tests should never time out.
|
| -
|
| -* A number followed by a unit abbreviation indicates an exact time. For example,
|
| - "1m" means a timeout of one minute, and "30s" means a timeout of thirty
|
| - seconds. Multiple numbers can be combined, as in "1m 30s".
|
| -
|
| -* A number followed by "x" indicates a multiple. This is applied to the default
|
| - value of 30s.
|
| -
|
| -```yaml
|
| -timeout: 1m
|
| -```
|
| -
|
| -### `verbose_trace`
|
| -
|
| -This boolean field controls whether or not stack traces caused by errors are
|
| -trimmed to remove internal stack frames. This includes frames from the Dart core
|
| -libraries, the [`stack_trace`][stack_trace] package, and the `test` package
|
| -itself. It defaults to `false`.
|
| -
|
| -[stack_trace]: https://pub.dartlang.org/packages/stack_trace
|
| -
|
| -```yaml
|
| -verbose_trace: true
|
| -```
|
| -
|
| -### `js_trace`
|
| -
|
| -This boolean field controls whether or not stack traces caused by errors that
|
| -occur while running Dart compiled to JS are converted back to Dart style. This
|
| -conversion uses the source map generated by `dart2js` to approximate the
|
| -original Dart line, column, and in some cases member name for each stack frame.
|
| -It defaults to `false`.
|
| -
|
| -```yaml
|
| -js_trace: true
|
| -```
|
| -
|
| -### `skip`
|
| -
|
| -This field controls whether or not tests are skipped. It's usually applied to
|
| -[specific tags](#configuring-tags) rather than used at the top level. Like the
|
| -`skip` parameter for [`test()`][test], it can either be a boolean indicating
|
| -whether the tests are skipped or a string indicating the reason they're skipped.
|
| -
|
| -[test]: https://www.dartdocs.org/documentation/test/0.12.10+2/test/test.html
|
| -
|
| -```yaml
|
| -tags:
|
| - chrome:
|
| - skip: "Our Chrome launcher is busted. See issue 1234."
|
| -```
|
| -
|
| -### `test_on`
|
| -
|
| -This field declares which platforms a test supports. It takes a
|
| -[platform selector][platform selectors] and only allows tests to run on
|
| -platforms that match the selector. It's often used with
|
| -[specific tags](#configuring-tags) to ensure that certain features will only be
|
| -tested on supported platforms.
|
| -
|
| -[platform selectors]: https://github.com/dart-lang/test/blob/master/README.md#platform-selectors
|
| -
|
| -```yaml
|
| -tags:
|
| - # Internet Explorer doesn't support promises yet.
|
| - promises: {test_on: "browser && !ie"}
|
| -```
|
| -
|
| -The field can also be used at the top level of the configuration file to
|
| -indicate that the entire package only supports a particular platform. If someone
|
| -tries to run the tests on an unsupported platform, the runner will print a
|
| -warning and skip that platform.
|
| -
|
| -```yaml
|
| -# This package uses dart:io.
|
| -test_on: vm
|
| -```
|
| -
|
| -## Runner Configuration
|
| -
|
| -Unlike [test configuration](#test-configuration), runner configuration affects
|
| -the test runner as a whole rather than individual tests. It can only be used at
|
| -the top level of the configuration file.
|
| -
|
| -### `paths`
|
| -
|
| -This field indicates the default paths that the test runner should run. These
|
| -paths are usually directories, although single filenames may be used as well.
|
| -Paths must be relative, and they must be in URL format so that they're
|
| -compatible across operating systems. This defaults to `[test]`.
|
| -
|
| -```yaml
|
| -paths: [dart/test]
|
| -
|
| -paths:
|
| -- test/instantaneous
|
| -- test/fast
|
| -- test/middling
|
| -```
|
| -
|
| -### `filename`
|
| -
|
| -This field indicates the filename pattern that the test runner uses to find test
|
| -files in directories. All files in directories passed on the command line (or in
|
| -directories in [`paths`](#paths), if none are passed) whose basenames match this
|
| -pattern will be loaded and run as tests.
|
| -
|
| -This supports the full [glob syntax][]. However, since it's only compared
|
| -against a path's basename, path separators aren't especially useful. It defaults
|
| -to `"*_test.dart"`.
|
| -
|
| -```yaml
|
| -filename: "test_*.dart"
|
| -```
|
| -
|
| -[glob syntax]: https://github.com/dart-lang/glob#syntax
|
| -
|
| -### `names`
|
| -
|
| -This field causes the runner to only run tests whose names match the given
|
| -regular expressions. A test's name must match *all* regular expressions in
|
| -`names`, as well as containing all strings in [`plain_names`](#plain_names), in
|
| -order to be run.
|
| -
|
| -This is usually used in a [preset](#configuration-presets) to make it possible
|
| -to quickly select a given set of tests.
|
| -
|
| -```yaml
|
| -presets:
|
| - # Pass "-P chrome" to run only Chrome tests.
|
| - chrome:
|
| - names:
|
| - - "^browser:"
|
| - - "[Cc]hrome"
|
| -```
|
| -
|
| -### `plain_names`
|
| -
|
| -This field causes the runner to only run tests whose names contain the given
|
| -strings. A test's name must contain *all* strings in `plain_names`, as well as
|
| -matching all regular expressions in [`names`](#names), in order to be run.
|
| -
|
| -This is usually used in a [preset](#configuration-presets) to make it possible
|
| -to quickly select a given set of tests.
|
| -
|
| -```yaml
|
| -presets:
|
| - # Pass "-P ie" to run only Internet Explorer tests.
|
| - ie:
|
| - plain_names:
|
| - - "IE"
|
| - - "Internet Explorer"
|
| -```
|
| -
|
| -### `include_tags`
|
| -
|
| -This field causes the runner to only run tests whose tags match the given
|
| -[boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags)
|
| -are used, the exclusions take precedence.
|
| -
|
| -[boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/README.md
|
| -
|
| -This is usually used in a [preset](#configuration-preset) to make it possible to
|
| -quickly select a set of tests.
|
| -
|
| -```yaml
|
| -presets:
|
| - # Pass "-P windowless" to run tests that don't open browser windows.
|
| - windowless:
|
| - include_tags: !browser || content-shell
|
| -```
|
| -
|
| -### `exclude_tags`
|
| -
|
| -This field causes the runner not to run tests whose tags match the given
|
| -[boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags`
|
| -are used, the exclusions take precedence.
|
| -
|
| -This is usually used in a [preset](#configuration-preset) to make it possible to
|
| -quickly select a set of tests.
|
| -
|
| -```yaml
|
| -presets:
|
| - # Pass "-P windowless" to run tests that don't open browser windows.
|
| - windowless:
|
| - exclude_tags: browser && !content-shell
|
| -```
|
| -
|
| -### `platforms`
|
| -
|
| -This field indicates which platforms tests should run on by default. It allows
|
| -the same platform identifiers that can be passed to `--platform`. If multiple
|
| -platforms are included, the test runner will default to running tests on all of
|
| -them. This defaults to `[vm]`.
|
| -
|
| -```yaml
|
| -platforms: [content_shell]
|
| -
|
| -platforms:
|
| -- chrome
|
| -- firefox
|
| -```
|
| -
|
| -### `concurrency`
|
| -
|
| -This field indicates the default number of test suites to run in parallel. More
|
| -parallelism can improve the overall speed of running tests up to a point, but
|
| -eventually it just adds more memory overhead without any performance gain. This
|
| -defaults to approximately half the number of processors on the current machine.
|
| -If it's set to 1, only one test suite will run at a time.
|
| -
|
| -```yaml
|
| -concurrency: 3
|
| -```
|
| -
|
| -### `pause_after_load`
|
| -
|
| -This field indicates that the test runner should pause for debugging after each
|
| -test suite is loaded but before its tests are executed. If it's set,
|
| -[`concurrency`](#concurrency) is automatically set to 1 and
|
| -[`timeout`](#timeout) is automatically set to `none`.
|
| -
|
| -This is usually used in a [preset](#configuration-presets).
|
| -
|
| -```yaml
|
| -presets:
|
| - # Pass "-P debug" to enable debugging configuration
|
| - debug:
|
| - pause_after_load: true
|
| - exclude_tags: undebuggable
|
| - reporter: expanded
|
| -```
|
| -
|
| -### `pub_serve`
|
| -
|
| -This field indicates that the test runner should run against a `pub serve`
|
| -instance by default, and provides the port number for that instance. Note that
|
| -if there is no `pub serve` instance running at that port, running the tests will
|
| -fail by default.
|
| -
|
| -```yaml
|
| -pub_serve: 8081
|
| -```
|
| -
|
| -### `reporter`
|
| -
|
| -This field indicates the default reporter to use. It may be set to "compact",
|
| -"expanded", or "json" (although why anyone would want to default to JSON is
|
| -beyond me). It defaults to "expanded" on Windows and "compact" everywhere else.
|
| -
|
| -```yaml
|
| -reporter: expanded
|
| -```
|
| -
|
| -## Configuring Tags
|
| -
|
| -### `tags`
|
| -
|
| -The `tag` field can be used to apply [test configuration](#test-configuration)
|
| -to all tests [with a given tag][tagging tests] or set of tags. It takes a map
|
| -from tag selectors to configuration maps. These configuration maps are just like
|
| -the top level of the configuration file, except that they may not contain
|
| -[runner configuration](#runner-configuration).
|
| -
|
| -[tagging tests]: https://github.com/dart-lang/test/blob/master/README.md#tagging-tests
|
| -
|
| -```yaml
|
| -tags:
|
| - # Integration tests need more time to run.
|
| - integration:
|
| - timeout: 1m
|
| -```
|
| -
|
| -Tags may also have no configuration associated with them. The test runner prints
|
| -a warning whenever it encounters a tag it doesn't recognize, and this the best
|
| -way to tell it that a tag exists.
|
| -
|
| -```yaml
|
| -# We occasionally want to use --tags or --exclude-tags on these tags.
|
| -tags:
|
| - # A test that spawns a browser.
|
| - browser:
|
| -
|
| - # A test that needs Ruby installed.
|
| - ruby:
|
| -```
|
| -
|
| -You can also use [boolean selector syntax][] to define configuration that
|
| -involves multiple tags. For example:
|
| -
|
| -[boolean selector syntax]: https://github.com/dart-lang/boolean_selector/blob/master/README.md
|
| -
|
| -```yaml
|
| -tags:
|
| - # Tests that invoke sub-processes tend to be a little slower.
|
| - ruby || python:
|
| - timeout: 1.5x
|
| -```
|
| -
|
| -Tag configuration is applied at whatever level the tag appears—so if a group is
|
| -tagged as `integration`, its timeout will take precedence over the suite's
|
| -timeout but not any tests'. If the group itself had a timeout declared, the
|
| -group's explicit timeout would take precedence over the tag.
|
| -
|
| -If multiple tags appear at the same level, and they have conflicting
|
| -configurations, the test runner *does not guarantee* what order they'll be
|
| -resolved in. In practice, conflicting configuration is pretty unlikely and it's
|
| -easy to just explicitly specify what you want on the test itself.
|
| -
|
| -This field counts as [test configuration](#test-configuration).
|
| -
|
| -### `add_tags`
|
| -
|
| -This field adds additional tags. It's technically
|
| -[test configuration](#test-configuration), but it's usually used in more
|
| -specific contexts. For example, when included in a tag's configuration, it can
|
| -be used to enable tag inheritance, where adding one tag implicitly adds another
|
| -as well. It takes a list of tag name strings.
|
| -
|
| -```yaml
|
| -tags:
|
| - # Any test that spawns a browser.
|
| - browser:
|
| - timeout: 2x
|
| -
|
| - # Tests that spawn specific browsers. These automatically get the browser tag
|
| - # as well.
|
| - chrome: {add_tags: [browser]}
|
| - firefox: {add_tags: [browser]}
|
| - safari: {add_tags: [browser]}
|
| - ie: {add_tags: [browser]}
|
| -```
|
| -
|
| -## Configuring Platforms
|
| -
|
| -There are two different kinds of platform configuration.
|
| -[Operating system configuration](#on_os) cares about the operating system on
|
| -which test runner is running. It sets global configuration for the runner on
|
| -particular OSes. [Test platform configuration](#on_platform), on the other hand,
|
| -cares about the platform the *test* is running on (like the
|
| -[`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular
|
| -tests that are running on matching platforms.
|
| -
|
| -[@OnPlatform]: https://github.com/dart-lang/test/blob/master/README.md#platform-specific-configuration
|
| -
|
| -### `on_os`
|
| -
|
| -This field applies configuration when specific operating systems are being used.
|
| -It takes a map from operating system identifiers (the same ones that are used in
|
| -[platform selectors][]) to configuration maps that are applied on those
|
| -operating systems. These configuration maps are just like the top level of the
|
| -configuration file, and allow any fields that may be used in the context where
|
| -`on_os` was used.
|
| -
|
| -```yaml
|
| -on_os:
|
| - windows:
|
| - # Both of these are the defaults anyway, but let's be explicit about it.
|
| - color: false
|
| - runner: expanded
|
| -
|
| - # My Windows machine is real slow.
|
| - timeout: 2x
|
| -
|
| - # My Linux machine has SO MUCH RAM.
|
| - linux:
|
| - concurrency: 500
|
| -```
|
| -
|
| -This field counts as [test configuration](#test-configuration). If it's used in
|
| -a context that only allows test configuration, it may only contain test
|
| -configuration.
|
| -
|
| -### `on_platform`
|
| -
|
| -This field applies configuration to tests that are run on specific platforms. It
|
| -takes a map from [platform selectors][] to configuration maps that are applied
|
| -to tests run on those platforms. These configuration maps are just like the top
|
| -level of the configuration file, except that they may not contain
|
| -[runner configuration](#runner-configuration).
|
| -
|
| -```yaml
|
| -# Our code is kind of slow on Blink and WebKit.
|
| -on_platform:
|
| - chrome || safari: {timeout: 2x}
|
| -```
|
| -
|
| -**Note**: operating system names that appear in `on_platform` refer to tests
|
| -that are run on the Dart VM under that operating system. To configure all tests
|
| -when running on a particular operating system, use [`on_os`](#on_os) instead.
|
| -
|
| -This field counts as [test configuration](#test-configuration).
|
| -
|
| -## Configuration Presets
|
| -
|
| -*Presets* are collections of configuration that can be explicitly selected on
|
| -the command-line. They're useful for quickly selecting options that are
|
| -frequently used together, for providing special configuration for continuous
|
| -integration systems, and for defining more complex logic than can be expressed
|
| -directly using command-line arguments.
|
| -
|
| -Presets can be selected on the command line using the `--preset` or `-P` flag.
|
| -Any number of presets can be selected this way; if they conflict, the last one
|
| -selected wins. Only presets that are defined in the configuration file may be
|
| -selected.
|
| -
|
| -### `presets`
|
| -
|
| -This field defines which presets are available to select. It takes a map from
|
| -preset names to configuration maps that are applied when those presets are
|
| -selected. These configuration maps are just like the top level of the
|
| -configuration file, and allow any fields that may be used in the context where
|
| -`presets` was used.
|
| -
|
| -```yaml
|
| -presets:
|
| - # Use this when you need completely un-munged stack traces.
|
| - debug:
|
| - verbose_trace: false
|
| - js_trace: true
|
| -
|
| - # Shortcut for running only browser tests.
|
| - browser:
|
| - paths:
|
| - - test/runner/browser
|
| - - test/runner/pub_serve_test.dart
|
| -```
|
| -
|
| -The `presets` field counts as [test configuration](#test-configuration). It can
|
| -be useful to use it in combination with other fields for advanced preset
|
| -behavior.
|
| -
|
| -```yaml
|
| -tags:
|
| - chrome:
|
| - skip: "Our Chrome launcher is busted. See issue 1234."
|
| -
|
| - # Pass -P force to verify that the launcher is still busted.
|
| - presets: {force: {skip: false}}
|
| -```
|
| -
|
| -### `add_presets`
|
| -
|
| -This field selects additional presets. It's technically
|
| -[runner configuration](#runner-configuration), but it's usually used in more
|
| -specific contexts. For example, when included in a preset's configuration, it
|
| -can be used to enable preset inheritance, where selecting one preset implicitly
|
| -selects another as well. It takes a list of preset name strings.
|
| -
|
| -```yaml
|
| -presets:
|
| - # Shortcut for running only browser tests.
|
| - browser:
|
| - paths: [test/runner/browser]
|
| -
|
| - # Shortcut for running only Chrome tests.
|
| - chrome:
|
| - filename: "chrome_*_test.dart"
|
| - add_presets: [browser]
|
| -```
|
|
|