| OLD | NEW |
|---|
| (Empty) |
| 1 JSON Reporter Protocol | |
| 2 ====================== | |
| 3 | |
| 4 The test runner supports a JSON reporter which provides a machine-readable | |
| 5 representation of the test runner's progress. This reporter is intended for use | |
| 6 by IDEs and other tools to present a custom view of the test runner's operation | |
| 7 without needing to parse output intended for humans. | |
| 8 | |
| 9 Note that the test runner is highly asynchronous, and users of this protocol | |
| 10 shouldn't make assumptions about the ordering of events beyond what's explicitly | |
| 11 specified in this document. It's possible for events from multiple tests to be | |
| 12 intertwined, for a single test to emit an error after it completed successfully, | |
| 13 and so on. | |
| 14 | |
| 15 ## Usage | |
| 16 | |
| 17 Pass the `--reporter json` command-line flag to the test runner to activate the | |
| 18 JSON reporter. | |
| 19 | |
| 20 pub run test --reporter json <path-to-test-file> | |
| 21 | |
| 22 The JSON stream will be emitted via standard output. It will be a stream of JSON | |
| 23 objects, separated by newlines. | |
| 24 | |
| 25 See `json_reporter.schema.json` for a formal description of the protocol schema. | |
| 26 See `test/runner/json_reporter_test.dart` for some sample output. | |
| 27 | |
| 28 ## Compatibility | |
| 29 | |
| 30 The protocol emitted by the JSON reporter is considered part of the public API | |
| 31 of the `test` package, and is subject to its [semantic versioning][semver] | |
| 32 restrictions. In particular: | |
| 33 | |
| 34 [semver]: https://www.dartlang.org/tools/pub/versioning.html#semantic-versions | |
| 35 | |
| 36 * No new feature will be added to the protocol without increasing the test | |
| 37 package's minor version number. | |
| 38 | |
| 39 * No breaking change will be made to the protocol without increasing the test | |
| 40 package's major version number. | |
| 41 | |
| 42 The following changes are not considered breaking. This is not necessarily a | |
| 43 comprehensive list. | |
| 44 | |
| 45 * Adding a new attribute to an existing object. | |
| 46 | |
| 47 * Adding a new type of any object with a `type` parameter. | |
| 48 | |
| 49 * Adding new test state values. | |
| 50 | |
| 51 ## Reading this Document | |
| 52 | |
| 53 Each major type of JSON object used by the protocol is described by a class. | |
| 54 Classes have names which are referred to in this document, but are not used as | |
| 55 part of the protocol. Classes have typed attributes, which refer to the types | |
| 56 and names of attributes in the JSON objects. If an attribute's type is another | |
| 57 class, that refers to a nested object. The special type `List<...>` indicates a | |
| 58 JSON list of the given type. | |
| 59 | |
| 60 Classes can "extend" one another, meaning that the subclass has all the | |
| 61 attributes of the superclass. Concrete subclasses can be distinguished by the | |
| 62 specific value of their `type` attribute. Classes may be abstract, indicating | |
| 63 that only their subclasses will ever be used. | |
| 64 | |
| 65 ## Events | |
| 66 | |
| 67 ### Event | |
| 68 | |
| 69 ``` | |
| 70 abstract class Event { | |
| 71 // The type of the event. | |
| 72 // | |
| 73 // This is always one of the subclass types listed below. | |
| 74 String type; | |
| 75 | |
| 76 // The time (in milliseconds) that has elapsed since the test runner started. | |
| 77 int time; | |
| 78 } | |
| 79 ``` | |
| 80 | |
| 81 This is the root class of the protocol. All root-level objects emitted by the | |
| 82 JSON reporter will be subclasses of `Event`. | |
| 83 | |
| 84 ### StartEvent | |
| 85 | |
| 86 ``` | |
| 87 class StartEvent extends Event { | |
| 88 String type = "start"; | |
| 89 | |
| 90 // The version of the JSON reporter protocol being used. | |
| 91 // | |
| 92 // This is a semantic version, but it reflects only the version of the | |
| 93 // protocol—it's not identical to the version of the test runner itself. | |
| 94 String protocolVersion; | |
| 95 | |
| 96 // The version of the test runner being used. | |
| 97 String runnerVersion; | |
| 98 } | |
| 99 ``` | |
| 100 | |
| 101 A single start event is emitted before any other events. It indicates that the | |
| 102 test runner has started running. | |
| 103 | |
| 104 ### AllSuitesEvent | |
| 105 | |
| 106 ``` | |
| 107 class AllSuitesEvent { | |
| 108 String type = "allSuites"; | |
| 109 | |
| 110 /// The total number of suites that will be loaded. | |
| 111 int count; | |
| 112 } | |
| 113 ``` | |
| 114 | |
| 115 A single suite count event is emitted once the test runner knows the total | |
| 116 number of suites that will be loaded over the course of the test run. Because | |
| 117 this is determined asynchronously, its position relative to other events (except | |
| 118 `StartEvent`) is not guaranteed. | |
| 119 | |
| 120 ### SuiteEvent | |
| 121 | |
| 122 ``` | |
| 123 class SuiteEvent extends Event { | |
| 124 String type = "suite"; | |
| 125 | |
| 126 /// Metadata about the suite. | |
| 127 Suite suite; | |
| 128 } | |
| 129 ``` | |
| 130 | |
| 131 A suite event is emitted before any `GroupEvent`s for groups in a given test | |
| 132 suite. This is the only event that contains the full metadata about a suite; | |
| 133 future events will refer to the suite by its opaque ID. | |
| 134 | |
| 135 ### GroupEvent | |
| 136 | |
| 137 ``` | |
| 138 class GroupEvent extends Event { | |
| 139 String type = "group"; | |
| 140 | |
| 141 /// Metadata about the group. | |
| 142 Group group; | |
| 143 } | |
| 144 ``` | |
| 145 | |
| 146 A group event is emitted before any `TestStartEvent`s for tests in a given | |
| 147 group. This is the only event that contains the full metadata about a group; | |
| 148 future events will refer to the group by its opaque ID. | |
| 149 | |
| 150 This includes the implicit group at the root of each suite, which has a `null` | |
| 151 name. However, it does *not* include implicit groups for the virtual suites | |
| 152 generated to represent loading test files. | |
| 153 | |
| 154 The group should be considered skipped if `group.metadata.skip` is `true`. When | |
| 155 a group is skipped, a single `TestStartEvent` will be emitted for a test within | |
| 156 that group that will also be skipped. | |
| 157 | |
| 158 ### TestStartEvent | |
| 159 | |
| 160 ``` | |
| 161 class TestStartEvent extends Event { | |
| 162 String type = "testStart"; | |
| 163 | |
| 164 // Metadata about the test that started. | |
| 165 Test test; | |
| 166 } | |
| 167 ``` | |
| 168 | |
| 169 An event emitted when a test begins running. This is the only event that | |
| 170 contains the full metadata about a test; future events will refer to the test by | |
| 171 its opaque ID. | |
| 172 | |
| 173 The test should be considered skipped if `test.metadata.skip` is `true`. | |
| 174 | |
| 175 ### PrintEvent | |
| 176 | |
| 177 ``` | |
| 178 class PrintEvent extends Event { | |
| 179 String type = "print"; | |
| 180 | |
| 181 // The ID of the test that printed a message. | |
| 182 int testID; | |
| 183 | |
| 184 // The message that was printed. | |
| 185 String message; | |
| 186 } | |
| 187 ``` | |
| 188 | |
| 189 A `PrintEvent` indicates that a test called `print()` and wishes to display | |
| 190 output. | |
| 191 | |
| 192 ### ErrorEvent | |
| 193 | |
| 194 ``` | |
| 195 class ErrorEvent extends Event { | |
| 196 String type = "error"; | |
| 197 | |
| 198 // The ID of the test that experienced the error. | |
| 199 int testID; | |
| 200 | |
| 201 // The result of calling toString() on the error object. | |
| 202 String error; | |
| 203 | |
| 204 // The error's stack trace, in the stack_trace package format. | |
| 205 String stackTrace; | |
| 206 | |
| 207 // Whether the error was a TestFailure. | |
| 208 bool isFailure; | |
| 209 } | |
| 210 ``` | |
| 211 | |
| 212 A `ErrorEvent` indicates that a test encountered an uncaught error. Note | |
| 213 that this may happen even after the test has completed, in which case it should | |
| 214 be considered to have failed. | |
| 215 | |
| 216 If a test is asynchronous, it may encounter multiple errors, which will result | |
| 217 in multiple `ErrorEvent`s. | |
| 218 | |
| 219 ### TestDoneEvent | |
| 220 | |
| 221 ``` | |
| 222 class TestDoneEvent extends Event { | |
| 223 String type = "testDone"; | |
| 224 | |
| 225 // The ID of the test that completed. | |
| 226 int testID; | |
| 227 | |
| 228 // The result of the test. | |
| 229 String result; | |
| 230 | |
| 231 // Whether the test's result should be hidden. | |
| 232 bool hidden; | |
| 233 } | |
| 234 ``` | |
| 235 | |
| 236 An event emitted when a test completes. The `result` attribute indicates the | |
| 237 result of the test: | |
| 238 | |
| 239 * `"success"` if the test had no errors. | |
| 240 | |
| 241 * `"failure"` if the test had a `TestFailure` but no other errors. | |
| 242 | |
| 243 * `"error"` if the test had an error other than a `TestFailure`. | |
| 244 | |
| 245 If the test encountered an error, the `TestDoneEvent` will be emitted after the | |
| 246 corresponding `ErrorEvent`. | |
| 247 | |
| 248 The `hidden` attribute indicates that the test's result should be hidden and not | |
| 249 counted towards the total number of tests run for the suite. This is true for | |
| 250 virtual tests created for loading test suites, `setUpAll()`, and | |
| 251 `tearDownAll()`. Only successful tests will be hidden. | |
| 252 | |
| 253 Note that it's possible for a test to encounter an error after completing. In | |
| 254 that case, it should be considered to have failed, but no additional | |
| 255 `TestDoneEvent` will be emitted. If a previously-hidden test encounters an | |
| 256 error after completing, it should be made visible. | |
| 257 | |
| 258 ### DoneEvent | |
| 259 | |
| 260 ``` | |
| 261 class DoneEvent extends Event { | |
| 262 String type = "done"; | |
| 263 | |
| 264 // Whether all tests succeeded (or were skipped). | |
| 265 bool success; | |
| 266 } | |
| 267 ``` | |
| 268 | |
| 269 An event indicating the result of the entire test run. This will be the final | |
| 270 event emitted by the reporter. | |
| 271 | |
| 272 ## Other Classes | |
| 273 | |
| 274 ### Test | |
| 275 | |
| 276 ``` | |
| 277 class Test { | |
| 278 // An opaque ID for the test. | |
| 279 int id; | |
| 280 | |
| 281 // The name of the test, including prefixes from any containing groups. | |
| 282 String name; | |
| 283 | |
| 284 // The ID of the suite containing this test. | |
| 285 int suiteID; | |
| 286 | |
| 287 // The IDs of groups containing this test, in order from outermost to | |
| 288 // innermost. | |
| 289 List<int> groupIDs; | |
| 290 | |
| 291 // The test's metadata, including metadata from any containing groups. | |
| 292 Metadata metadata; | |
| 293 } | |
| 294 ``` | |
| 295 | |
| 296 A single test case. The test's ID is unique in the context of this test run. | |
| 297 It's used elsewhere in the protocol to refer to this test without including its | |
| 298 full representation. | |
| 299 | |
| 300 Most tests will have at least one group ID, representing the implicit root | |
| 301 group. However, some may not; these should be treated as having no group | |
| 302 metadata. | |
| 303 | |
| 304 ### Suite | |
| 305 | |
| 306 ``` | |
| 307 class Suite { | |
| 308 // An opaque ID for the group. | |
| 309 int id; | |
| 310 | |
| 311 // The platform on which the suite is running. | |
| 312 String? platform; | |
| 313 | |
| 314 // The path to the suite's file. | |
| 315 String path; | |
| 316 } | |
| 317 ``` | |
| 318 | |
| 319 A test suite corresponding to a loaded test file. The suite's ID is unique in | |
| 320 the context of this test run. It's used elsewhere in the protocol to refer to | |
| 321 this suite without including its full representation. | |
| 322 | |
| 323 A suite's platform is one of the platforms that can be passed to the | |
| 324 `--platform` option, or `null` if there is no platform (for example if the file | |
| 325 doesn't exist at all). Its path is relative to the root of the current package. | |
| 326 | |
| 327 Suites don't include their own metadata. Instead, that metadata is present on | |
| 328 the root-level group. | |
| 329 | |
| 330 ### Group | |
| 331 | |
| 332 ``` | |
| 333 class Group { | |
| 334 // An opaque ID for the group. | |
| 335 int id; | |
| 336 | |
| 337 // The name of the group, including prefixes from any containing groups. | |
| 338 String? name; | |
| 339 | |
| 340 // The ID of the suite containing this group. | |
| 341 int suiteID; | |
| 342 | |
| 343 // The ID of the group's parent group, unless it's the root group. | |
| 344 int? parentID; | |
| 345 | |
| 346 // The group's metadata, including metadata from any containing groups. | |
| 347 Metadata metadata; | |
| 348 | |
| 349 // The number of tests (recursively) within this group. | |
| 350 int testCount; | |
| 351 } | |
| 352 ``` | |
| 353 | |
| 354 A group containing test cases. The group's ID is unique in the context of this | |
| 355 test run. It's used elsewhere in the protocol to refer to this group without | |
| 356 including its full representation. | |
| 357 | |
| 358 The implicit group at the root of each test suite has `null` `name` and | |
| 359 `parentID` attributes. | |
| 360 | |
| 361 ### Metadata | |
| 362 | |
| 363 ``` | |
| 364 class Metadata { | |
| 365 // Whether the test case will be skipped by the test runner. | |
| 366 bool skip; | |
| 367 | |
| 368 // The reason the test case is skipped, if the user provided it. | |
| 369 String? skipReason; | |
| 370 } | |
| 371 ``` | |
| 372 | |
| 373 The metadata attached to a test by a user. | |
| OLD | NEW |
|---|
Chromium Code Reviews
Issue 1661873002:
Improve documentation organization. (Closed)
Base URL: git@github.com:dart-lang/test@master