| Index: json_reporter.md
|
| diff --git a/json_reporter.md b/json_reporter.md
|
| deleted file mode 100644
|
| index 981cf6375fba20115e0d105fabfa038b6474aed6..0000000000000000000000000000000000000000
|
| --- a/json_reporter.md
|
| +++ /dev/null
|
| @@ -1,373 +0,0 @@
|
| -JSON Reporter Protocol
|
| -======================
|
| -
|
| -The test runner supports a JSON reporter which provides a machine-readable
|
| -representation of the test runner's progress. This reporter is intended for use
|
| -by IDEs and other tools to present a custom view of the test runner's operation
|
| -without needing to parse output intended for humans.
|
| -
|
| -Note that the test runner is highly asynchronous, and users of this protocol
|
| -shouldn't make assumptions about the ordering of events beyond what's explicitly
|
| -specified in this document. It's possible for events from multiple tests to be
|
| -intertwined, for a single test to emit an error after it completed successfully,
|
| -and so on.
|
| -
|
| -## Usage
|
| -
|
| -Pass the `--reporter json` command-line flag to the test runner to activate the
|
| -JSON reporter.
|
| -
|
| - pub run test --reporter json <path-to-test-file>
|
| -
|
| -The JSON stream will be emitted via standard output. It will be a stream of JSON
|
| -objects, separated by newlines.
|
| -
|
| -See `json_reporter.schema.json` for a formal description of the protocol schema.
|
| -See `test/runner/json_reporter_test.dart` for some sample output.
|
| -
|
| -## Compatibility
|
| -
|
| -The protocol emitted by the JSON reporter is considered part of the public API
|
| -of the `test` package, and is subject to its [semantic versioning][semver]
|
| -restrictions. In particular:
|
| -
|
| -[semver]: https://www.dartlang.org/tools/pub/versioning.html#semantic-versions
|
| -
|
| -* No new feature will be added to the protocol without increasing the test
|
| - package's minor version number.
|
| -
|
| -* No breaking change will be made to the protocol without increasing the test
|
| - package's major version number.
|
| -
|
| -The following changes are not considered breaking. This is not necessarily a
|
| -comprehensive list.
|
| -
|
| -* Adding a new attribute to an existing object.
|
| -
|
| -* Adding a new type of any object with a `type` parameter.
|
| -
|
| -* Adding new test state values.
|
| -
|
| -## Reading this Document
|
| -
|
| -Each major type of JSON object used by the protocol is described by a class.
|
| -Classes have names which are referred to in this document, but are not used as
|
| -part of the protocol. Classes have typed attributes, which refer to the types
|
| -and names of attributes in the JSON objects. If an attribute's type is another
|
| -class, that refers to a nested object. The special type `List<...>` indicates a
|
| -JSON list of the given type.
|
| -
|
| -Classes can "extend" one another, meaning that the subclass has all the
|
| -attributes of the superclass. Concrete subclasses can be distinguished by the
|
| -specific value of their `type` attribute. Classes may be abstract, indicating
|
| -that only their subclasses will ever be used.
|
| -
|
| -## Events
|
| -
|
| -### Event
|
| -
|
| -```
|
| -abstract class Event {
|
| - // The type of the event.
|
| - //
|
| - // This is always one of the subclass types listed below.
|
| - String type;
|
| -
|
| - // The time (in milliseconds) that has elapsed since the test runner started.
|
| - int time;
|
| -}
|
| -```
|
| -
|
| -This is the root class of the protocol. All root-level objects emitted by the
|
| -JSON reporter will be subclasses of `Event`.
|
| -
|
| -### StartEvent
|
| -
|
| -```
|
| -class StartEvent extends Event {
|
| - String type = "start";
|
| -
|
| - // The version of the JSON reporter protocol being used.
|
| - //
|
| - // This is a semantic version, but it reflects only the version of the
|
| - // protocol—it's not identical to the version of the test runner itself.
|
| - String protocolVersion;
|
| -
|
| - // The version of the test runner being used.
|
| - String runnerVersion;
|
| -}
|
| -```
|
| -
|
| -A single start event is emitted before any other events. It indicates that the
|
| -test runner has started running.
|
| -
|
| -### AllSuitesEvent
|
| -
|
| -```
|
| -class AllSuitesEvent {
|
| - String type = "allSuites";
|
| -
|
| - /// The total number of suites that will be loaded.
|
| - int count;
|
| -}
|
| -```
|
| -
|
| -A single suite count event is emitted once the test runner knows the total
|
| -number of suites that will be loaded over the course of the test run. Because
|
| -this is determined asynchronously, its position relative to other events (except
|
| -`StartEvent`) is not guaranteed.
|
| -
|
| -### SuiteEvent
|
| -
|
| -```
|
| -class SuiteEvent extends Event {
|
| - String type = "suite";
|
| -
|
| - /// Metadata about the suite.
|
| - Suite suite;
|
| -}
|
| -```
|
| -
|
| -A suite event is emitted before any `GroupEvent`s for groups in a given test
|
| -suite. This is the only event that contains the full metadata about a suite;
|
| -future events will refer to the suite by its opaque ID.
|
| -
|
| -### GroupEvent
|
| -
|
| -```
|
| -class GroupEvent extends Event {
|
| - String type = "group";
|
| -
|
| - /// Metadata about the group.
|
| - Group group;
|
| -}
|
| -```
|
| -
|
| -A group event is emitted before any `TestStartEvent`s for tests in a given
|
| -group. This is the only event that contains the full metadata about a group;
|
| -future events will refer to the group by its opaque ID.
|
| -
|
| -This includes the implicit group at the root of each suite, which has a `null`
|
| -name. However, it does *not* include implicit groups for the virtual suites
|
| -generated to represent loading test files.
|
| -
|
| -The group should be considered skipped if `group.metadata.skip` is `true`. When
|
| -a group is skipped, a single `TestStartEvent` will be emitted for a test within
|
| -that group that will also be skipped.
|
| -
|
| -### TestStartEvent
|
| -
|
| -```
|
| -class TestStartEvent extends Event {
|
| - String type = "testStart";
|
| -
|
| - // Metadata about the test that started.
|
| - Test test;
|
| -}
|
| -```
|
| -
|
| -An event emitted when a test begins running. This is the only event that
|
| -contains the full metadata about a test; future events will refer to the test by
|
| -its opaque ID.
|
| -
|
| -The test should be considered skipped if `test.metadata.skip` is `true`.
|
| -
|
| -### PrintEvent
|
| -
|
| -```
|
| -class PrintEvent extends Event {
|
| - String type = "print";
|
| -
|
| - // The ID of the test that printed a message.
|
| - int testID;
|
| -
|
| - // The message that was printed.
|
| - String message;
|
| -}
|
| -```
|
| -
|
| -A `PrintEvent` indicates that a test called `print()` and wishes to display
|
| -output.
|
| -
|
| -### ErrorEvent
|
| -
|
| -```
|
| -class ErrorEvent extends Event {
|
| - String type = "error";
|
| -
|
| - // The ID of the test that experienced the error.
|
| - int testID;
|
| -
|
| - // The result of calling toString() on the error object.
|
| - String error;
|
| -
|
| - // The error's stack trace, in the stack_trace package format.
|
| - String stackTrace;
|
| -
|
| - // Whether the error was a TestFailure.
|
| - bool isFailure;
|
| -}
|
| -```
|
| -
|
| -A `ErrorEvent` indicates that a test encountered an uncaught error. Note
|
| -that this may happen even after the test has completed, in which case it should
|
| -be considered to have failed.
|
| -
|
| -If a test is asynchronous, it may encounter multiple errors, which will result
|
| -in multiple `ErrorEvent`s.
|
| -
|
| -### TestDoneEvent
|
| -
|
| -```
|
| -class TestDoneEvent extends Event {
|
| - String type = "testDone";
|
| -
|
| - // The ID of the test that completed.
|
| - int testID;
|
| -
|
| - // The result of the test.
|
| - String result;
|
| -
|
| - // Whether the test's result should be hidden.
|
| - bool hidden;
|
| -}
|
| -```
|
| -
|
| -An event emitted when a test completes. The `result` attribute indicates the
|
| -result of the test:
|
| -
|
| -* `"success"` if the test had no errors.
|
| -
|
| -* `"failure"` if the test had a `TestFailure` but no other errors.
|
| -
|
| -* `"error"` if the test had an error other than a `TestFailure`.
|
| -
|
| -If the test encountered an error, the `TestDoneEvent` will be emitted after the
|
| -corresponding `ErrorEvent`.
|
| -
|
| -The `hidden` attribute indicates that the test's result should be hidden and not
|
| -counted towards the total number of tests run for the suite. This is true for
|
| -virtual tests created for loading test suites, `setUpAll()`, and
|
| -`tearDownAll()`. Only successful tests will be hidden.
|
| -
|
| -Note that it's possible for a test to encounter an error after completing. In
|
| -that case, it should be considered to have failed, but no additional
|
| -`TestDoneEvent` will be emitted. If a previously-hidden test encounters an
|
| -error after completing, it should be made visible.
|
| -
|
| -### DoneEvent
|
| -
|
| -```
|
| -class DoneEvent extends Event {
|
| - String type = "done";
|
| -
|
| - // Whether all tests succeeded (or were skipped).
|
| - bool success;
|
| -}
|
| -```
|
| -
|
| -An event indicating the result of the entire test run. This will be the final
|
| -event emitted by the reporter.
|
| -
|
| -## Other Classes
|
| -
|
| -### Test
|
| -
|
| -```
|
| -class Test {
|
| - // An opaque ID for the test.
|
| - int id;
|
| -
|
| - // The name of the test, including prefixes from any containing groups.
|
| - String name;
|
| -
|
| - // The ID of the suite containing this test.
|
| - int suiteID;
|
| -
|
| - // The IDs of groups containing this test, in order from outermost to
|
| - // innermost.
|
| - List<int> groupIDs;
|
| -
|
| - // The test's metadata, including metadata from any containing groups.
|
| - Metadata metadata;
|
| -}
|
| -```
|
| -
|
| -A single test case. The test's ID is unique in the context of this test run.
|
| -It's used elsewhere in the protocol to refer to this test without including its
|
| -full representation.
|
| -
|
| -Most tests will have at least one group ID, representing the implicit root
|
| -group. However, some may not; these should be treated as having no group
|
| -metadata.
|
| -
|
| -### Suite
|
| -
|
| -```
|
| -class Suite {
|
| - // An opaque ID for the group.
|
| - int id;
|
| -
|
| - // The platform on which the suite is running.
|
| - String? platform;
|
| -
|
| - // The path to the suite's file.
|
| - String path;
|
| -}
|
| -```
|
| -
|
| -A test suite corresponding to a loaded test file. The suite's ID is unique in
|
| -the context of this test run. It's used elsewhere in the protocol to refer to
|
| -this suite without including its full representation.
|
| -
|
| -A suite's platform is one of the platforms that can be passed to the
|
| -`--platform` option, or `null` if there is no platform (for example if the file
|
| -doesn't exist at all). Its path is relative to the root of the current package.
|
| -
|
| -Suites don't include their own metadata. Instead, that metadata is present on
|
| -the root-level group.
|
| -
|
| -### Group
|
| -
|
| -```
|
| -class Group {
|
| - // An opaque ID for the group.
|
| - int id;
|
| -
|
| - // The name of the group, including prefixes from any containing groups.
|
| - String? name;
|
| -
|
| - // The ID of the suite containing this group.
|
| - int suiteID;
|
| -
|
| - // The ID of the group's parent group, unless it's the root group.
|
| - int? parentID;
|
| -
|
| - // The group's metadata, including metadata from any containing groups.
|
| - Metadata metadata;
|
| -
|
| - // The number of tests (recursively) within this group.
|
| - int testCount;
|
| -}
|
| -```
|
| -
|
| -A group containing test cases. The group's ID is unique in the context of this
|
| -test run. It's used elsewhere in the protocol to refer to this group without
|
| -including its full representation.
|
| -
|
| -The implicit group at the root of each test suite has `null` `name` and
|
| -`parentID` attributes.
|
| -
|
| -### Metadata
|
| -
|
| -```
|
| -class Metadata {
|
| - // Whether the test case will be skipped by the test runner.
|
| - bool skip;
|
| -
|
| - // The reason the test case is skipped, if the user provided it.
|
| - String? skipReason;
|
| -}
|
| -```
|
| -
|
| -The metadata attached to a test by a user.
|
|
|