OLD | NEW |
(Empty) | |
| 1 JSON Reporter Protocol |
| 2 ====================== |
| 3 |
| 4 The test runner supports a JSON reporter which provides a machine-readable |
| 5 representation of the test runner's progress. This reporter is intended for use |
| 6 by IDEs and other tools to present a custom view of the test runner's operation |
| 7 without needing to parse output intended for humans. |
| 8 |
| 9 ## Usage |
| 10 |
| 11 Pass the `--reporter json` command-line flag to the test runner to activate the |
| 12 JSON reporter. |
| 13 |
| 14 pub run test --reporter json <path-to-test-file> |
| 15 |
| 16 The JSON stream will be emitted via standard output. It will be a stream of JSON |
| 17 objects, separated by newlines. |
| 18 |
| 19 See `json_reporter.schema.json` for a formal description of the protocol schema. |
| 20 See `test/runner/json_reporter_test.dart` for some sample output. |
| 21 |
| 22 ## Compatibility |
| 23 |
| 24 The protocol emitted by the JSON reporter is considered part of the public API |
| 25 of the `test` package, and is subject to its [semantic versioning][semver] |
| 26 restrictions. In particular: |
| 27 |
| 28 [semver]: https://www.dartlang.org/tools/pub/versioning.html#semantic-versions |
| 29 |
| 30 * No new feature will be added to the protocol without increasing the test |
| 31 package's minor version number. |
| 32 |
| 33 * No breaking change will be made to the protocol without increasing the test |
| 34 package's major version number. |
| 35 |
| 36 The following changes are not considered breaking. This is not necessarily a |
| 37 comprehensive list. |
| 38 |
| 39 * Adding a new attribute to an existing object. |
| 40 |
| 41 * Adding a new type of any object with a `type` parameter. |
| 42 |
| 43 * Adding new test state values. |
| 44 |
| 45 ## Reading this Document |
| 46 |
| 47 Each major type of JSON object used by the protocol is described by a class. |
| 48 Classes have names which are referred to in this document, but are not used as |
| 49 part of the protocol. Classes have typed attributes, which refer to the types |
| 50 and names of attributes in the JSON objects. If an attribute's type is another |
| 51 class, that refers to a nested object. The special type `List<...>` indicates a |
| 52 JSON list of the given type. |
| 53 |
| 54 Classes can "extend" one another, meaning that the subclass has all the |
| 55 properties of the superclass. Concrete subclasses can be distinguished by the |
| 56 specific value of their `type` attribute. Classes may be abstract, indicating |
| 57 that only their subclasses will ever be used. |
| 58 |
| 59 ## Events |
| 60 |
| 61 ### Event |
| 62 |
| 63 ``` |
| 64 abstract class Event { |
| 65 // The type of the event. |
| 66 // |
| 67 // This is always one of the subclass types listed below. |
| 68 String type; |
| 69 |
| 70 // The time (in milliseconds) that has elapsed since the test runner started. |
| 71 int time; |
| 72 } |
| 73 ``` |
| 74 |
| 75 This is the root class of the protocol. All root-level objects emitted by the |
| 76 JSON reporter will be subclasses of `Event`. |
| 77 |
| 78 ### StartEvent |
| 79 |
| 80 ``` |
| 81 class StartEvent extends Event { |
| 82 String type = "start"; |
| 83 |
| 84 // The version of the JSON reporter protocol being used. |
| 85 // |
| 86 // This is a semantic version, but it reflects only the version of the |
| 87 // protocol—it's not identical to the version of the test runner itself. |
| 88 String protocolVersion; |
| 89 |
| 90 // The version of the test runner being used. |
| 91 String runnerVersion; |
| 92 } |
| 93 ``` |
| 94 |
| 95 A single start event is emitted before any other events. It indicates that the |
| 96 test runner has started running. |
| 97 |
| 98 ### TestStartEvent |
| 99 |
| 100 ``` |
| 101 class TestStartEvent extends Event { |
| 102 String type = "testStart"; |
| 103 |
| 104 // Metadata about the test that started. |
| 105 Test test; |
| 106 } |
| 107 ``` |
| 108 |
| 109 An event emitted when a test begins running. This is the only event that |
| 110 contains the full metadata about a test; future events will refer to the test by |
| 111 its opaque ID. |
| 112 |
| 113 The test should be considered skipped if `test.metadata.skip` is `true`. |
| 114 |
| 115 ### PrintEvent |
| 116 |
| 117 ``` |
| 118 class PrintEvent extends Event { |
| 119 String type = "print"; |
| 120 |
| 121 // The ID of the test that printed a message. |
| 122 int testID; |
| 123 |
| 124 // The message that was printed. |
| 125 String message; |
| 126 } |
| 127 ``` |
| 128 |
| 129 A `PrintEvent` indicates that a test called `print()` and wishes to display |
| 130 output. |
| 131 |
| 132 ### ErrorEvent |
| 133 |
| 134 ``` |
| 135 class ErrorEvent extends Event { |
| 136 String type = "error"; |
| 137 |
| 138 // The ID of the test that experienced the error. |
| 139 int testID; |
| 140 |
| 141 // The result of calling toString() on the error object. |
| 142 String error; |
| 143 |
| 144 // The error's stack trace, in the stack_trace package format. |
| 145 String stackTrace; |
| 146 |
| 147 // Whether the error was a TestFailure. |
| 148 bool isFailure; |
| 149 } |
| 150 ``` |
| 151 |
| 152 A `ErrorEvent` indicates that a test encountered an uncaught error. Note |
| 153 that this may happen even after the test has completed, in which case it should |
| 154 be considered to have failed. |
| 155 |
| 156 If a test is asynchronous, it may encounter multiple errors, which will result |
| 157 in multiple `ErrorEvent`s. |
| 158 |
| 159 ### TestDoneEvent |
| 160 |
| 161 ``` |
| 162 class TestDoneEvent extends Event { |
| 163 String type = "testDone"; |
| 164 |
| 165 // The ID of the test that completed. |
| 166 int testID; |
| 167 |
| 168 // The result of the test. |
| 169 String result; |
| 170 |
| 171 // Whether the test's result should be hidden. |
| 172 bool hidden; |
| 173 } |
| 174 ``` |
| 175 |
| 176 An event emitted when a test completes. The `result` attribute indicates the |
| 177 result of the test: |
| 178 |
| 179 * `"success"` if the test had no errors. |
| 180 |
| 181 * `"failure"` if the test had a `TestFailure` but no other errors. |
| 182 |
| 183 * `"error"` if the test had an error other than a `TestFailure`. |
| 184 |
| 185 If the test encountered an error, the `TestDoneEvent` will be emitted after the |
| 186 corresponding `ErrorEvent`. |
| 187 |
| 188 The `hidden` attribute indicates that the test's result should be hidden and not |
| 189 counted towards the total number of tests run for the suite. This is true for |
| 190 virtual tests created for loading test suites, `setUpAll()`, and |
| 191 `tearDownAll()`. Only successful tests will be hidden. |
| 192 |
| 193 Note that it's possible for a test to encounter an error after completing. In |
| 194 that case, it should be considered to have failed, but no additional |
| 195 `TestDoneEvent` will be emitted. If a previously-hidden test encounters an |
| 196 error after completing, it should be made visible. |
| 197 |
| 198 ### DoneEvent |
| 199 |
| 200 ``` |
| 201 class DoneEvent extends Event { |
| 202 String type = "done"; |
| 203 |
| 204 // Whether all tests succeeded (or were skipped). |
| 205 bool success; |
| 206 } |
| 207 ``` |
| 208 |
| 209 An event indicating the result of the entire test run. This will be the final |
| 210 event emitted by the reporter. |
| 211 |
| 212 ## Test Information |
| 213 |
| 214 ### Test |
| 215 |
| 216 ``` |
| 217 class Test { |
| 218 // An opaque ID for the test. |
| 219 int id; |
| 220 |
| 221 // The name of the test, including prefixes from any containing groups. |
| 222 String name; |
| 223 |
| 224 // The test's metadata, including metadata from any containing groups and the |
| 225 // test suite itself. |
| 226 Metadata metadata; |
| 227 } |
| 228 ``` |
| 229 |
| 230 A single test case. The test's ID is unique in the context of this test run. |
| 231 It's used elsewhere in the protocol to refer to this test without including its |
| 232 full representation. |
| 233 |
| 234 ### Metadata |
| 235 |
| 236 ``` |
| 237 class Metadata { |
| 238 // Whether the test case will be skipped by the test runner. |
| 239 bool skip; |
| 240 |
| 241 // The reason the test case is skipped, if the user provided it. |
| 242 String? skipReason; |
| 243 } |
| 244 ``` |
| 245 |
| 246 The metadata attached to a test by a user. |
OLD | NEW |