Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(123)

Side by Side Diff: doc/json_reporter.md

Issue 2091173002: Add metadata to messages emitted by live tests. (Closed) Base URL: git@github.com:dart-lang/test@master
Patch Set: Created 4 years, 6 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | json_reporter.schema.json » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 JSON Reporter Protocol 1 JSON Reporter Protocol
2 ====================== 2 ======================
3 3
4 The test runner supports a JSON reporter which provides a machine-readable 4 The test runner supports a JSON reporter which provides a machine-readable
5 representation of the test runner's progress. This reporter is intended for use 5 representation of the test runner's progress. This reporter is intended for use
6 by IDEs and other tools to present a custom view of the test runner's operation 6 by IDEs and other tools to present a custom view of the test runner's operation
7 without needing to parse output intended for humans. 7 without needing to parse output intended for humans.
8 8
9 Note that the test runner is highly asynchronous, and users of this protocol 9 Note that the test runner is highly asynchronous, and users of this protocol
10 shouldn't make assumptions about the ordering of events beyond what's explicitly 10 shouldn't make assumptions about the ordering of events beyond what's explicitly
(...skipping 135 matching lines...) Expand 10 before | Expand all | Expand 10 after
146 A group event is emitted before any `TestStartEvent`s for tests in a given 146 A group event is emitted before any `TestStartEvent`s for tests in a given
147 group. This is the only event that contains the full metadata about a group; 147 group. This is the only event that contains the full metadata about a group;
148 future events will refer to the group by its opaque ID. 148 future events will refer to the group by its opaque ID.
149 149
150 This includes the implicit group at the root of each suite, which has a `null` 150 This includes the implicit group at the root of each suite, which has a `null`
151 name. However, it does *not* include implicit groups for the virtual suites 151 name. However, it does *not* include implicit groups for the virtual suites
152 generated to represent loading test files. 152 generated to represent loading test files.
153 153
154 If the group is skipped, a single `TestStartEvent` will be emitted for a test 154 If the group is skipped, a single `TestStartEvent` will be emitted for a test
155 within the group, followed by a `TestDoneEvent` marked as skipped. The 155 within the group, followed by a `TestDoneEvent` marked as skipped. The
156 `group.metadata.skip` field should *not* be considered authoritative for 156 `group.metadata` field should *not* be used for determining whether a group is
157 determining whether a group is skipped. 157 skipped.
158 158
159 ### TestStartEvent 159 ### TestStartEvent
160 160
161 ``` 161 ```
162 class TestStartEvent extends Event { 162 class TestStartEvent extends Event {
163 String type = "testStart"; 163 String type = "testStart";
164 164
165 // Metadata about the test that started. 165 // Metadata about the test that started.
166 Test test; 166 Test test;
167 } 167 }
168 ``` 168 ```
169 169
170 An event emitted when a test begins running. This is the only event that 170 An event emitted when a test begins running. This is the only event that
171 contains the full metadata about a test; future events will refer to the test by 171 contains the full metadata about a test; future events will refer to the test by
172 its opaque ID. 172 its opaque ID.
173 173
174 If the test is skipped, its `TestDoneEvent` will have `skipped` set to `true`. 174 If the test is skipped, its `TestDoneEvent` will have `skipped` set to `true`.
175 The `test.metadata.skip` field should *not* be considered authoritative for 175 The `test.metadata` should *not* be used for determining whether a test is
176 determining whether a test is skipped. 176 skipped.
177 177
178 ### PrintEvent 178 ### MessageEvent
179 179
180 ``` 180 ```
181 class PrintEvent extends Event { 181 class MessageEvent extends Event {
182 String type = "print"; 182 String type = "print";
183 183
184 // The ID of the test that printed a message. 184 // The ID of the test that printed a message.
185 int testID; 185 int testID;
186 186
187 // The type of message being printed.
188 String messageType;
189
187 // The message that was printed. 190 // The message that was printed.
188 String message; 191 String message;
189 } 192 }
190 ``` 193 ```
191 194
192 A `PrintEvent` indicates that a test called `print()` and wishes to display 195 A `MessageEvent` indicates that a test emitted a message that should be
193 output. 196 displayed to the user. The `messageType` field indicates the precise type of
197 this message. Different message types should be visually distinguishable.
198
199 A message of type "print" comes from a user explicitly calling `print()`.
200
201 A message of type "skip" comes from a test, or a section of a test, being
202 skipped. A skip message shouldn't be considered the authoritative source that a
203 test was skipped; the `TestDoneEvent.skipped` field should be used instead.
194 204
195 ### ErrorEvent 205 ### ErrorEvent
196 206
197 ``` 207 ```
198 class ErrorEvent extends Event { 208 class ErrorEvent extends Event {
199 String type = "error"; 209 String type = "error";
200 210
201 // The ID of the test that experienced the error. 211 // The ID of the test that experienced the error.
202 int testID; 212 int testID;
203 213
(...skipping 83 matching lines...) Expand 10 before | Expand all | Expand 10 after
287 // The name of the test, including prefixes from any containing groups. 297 // The name of the test, including prefixes from any containing groups.
288 String name; 298 String name;
289 299
290 // The ID of the suite containing this test. 300 // The ID of the suite containing this test.
291 int suiteID; 301 int suiteID;
292 302
293 // The IDs of groups containing this test, in order from outermost to 303 // The IDs of groups containing this test, in order from outermost to
294 // innermost. 304 // innermost.
295 List<int> groupIDs; 305 List<int> groupIDs;
296 306
297 // The test's metadata, including metadata from any containing groups.
298 Metadata metadata;
299
300 // The (1-based) line on which the test was defined, or `null`. 307 // The (1-based) line on which the test was defined, or `null`.
301 int line; 308 int line;
302 309
303 // The (1-based) column on which the test was defined, or `null`. 310 // The (1-based) column on which the test was defined, or `null`.
304 int column; 311 int column;
305 312
306 // The URL for the file in which the test was defined, or `null`. 313 // The URL for the file in which the test was defined, or `null`.
307 String url; 314 String url;
315
316 // This field is deprecated and should not be used.
317 Metadata metadata;
308 } 318 }
309 ``` 319 ```
310 320
311 A single test case. The test's ID is unique in the context of this test run. 321 A single test case. The test's ID is unique in the context of this test run.
312 It's used elsewhere in the protocol to refer to this test without including its 322 It's used elsewhere in the protocol to refer to this test without including its
313 full representation. 323 full representation.
314 324
315 Most tests will have at least one group ID, representing the implicit root 325 Most tests will have at least one group ID, representing the implicit root
316 group. However, some may not; these should be treated as having no group 326 group. However, some may not; these should be treated as having no group
317 metadata. 327 metadata.
(...skipping 20 matching lines...) Expand all
338 348
339 A test suite corresponding to a loaded test file. The suite's ID is unique in 349 A test suite corresponding to a loaded test file. The suite's ID is unique in
340 the context of this test run. It's used elsewhere in the protocol to refer to 350 the context of this test run. It's used elsewhere in the protocol to refer to
341 this suite without including its full representation. 351 this suite without including its full representation.
342 352
343 A suite's platform is one of the platforms that can be passed to the 353 A suite's platform is one of the platforms that can be passed to the
344 `--platform` option, or `null` if there is no platform (for example if the file 354 `--platform` option, or `null` if there is no platform (for example if the file
345 doesn't exist at all). Its path is either absolute or relative to the root of 355 doesn't exist at all). Its path is either absolute or relative to the root of
346 the current package. 356 the current package.
347 357
348 Suites don't include their own metadata. Instead, that metadata is present on
349 the root-level group.
350
351 ### Group 358 ### Group
352 359
353 ``` 360 ```
354 class Group { 361 class Group {
355 // An opaque ID for the group. 362 // An opaque ID for the group.
356 int id; 363 int id;
357 364
358 // The name of the group, including prefixes from any containing groups. 365 // The name of the group, including prefixes from any containing groups.
359 String? name; 366 String? name;
360 367
361 // The ID of the suite containing this group. 368 // The ID of the suite containing this group.
362 int suiteID; 369 int suiteID;
363 370
364 // The ID of the group's parent group, unless it's the root group. 371 // The ID of the group's parent group, unless it's the root group.
365 int? parentID; 372 int? parentID;
366 373
367 // The group's metadata, including metadata from any containing groups.
368 Metadata metadata;
369
370 // The number of tests (recursively) within this group. 374 // The number of tests (recursively) within this group.
371 int testCount; 375 int testCount;
372 376
373 // The (1-based) line on which the group was defined, or `null`. 377 // The (1-based) line on which the group was defined, or `null`.
374 int line; 378 int line;
375 379
376 // The (1-based) column on which the group was defined, or `null`. 380 // The (1-based) column on which the group was defined, or `null`.
377 int column; 381 int column;
378 382
379 // The URL for the file in which the group was defined, or `null`. 383 // The URL for the file in which the group was defined, or `null`.
380 String url; 384 String url;
385
386 // This field is deprecated and should not be used.
387 Metadata metadata;
381 } 388 }
382 ``` 389 ```
383 390
384 A group containing test cases. The group's ID is unique in the context of this 391 A group containing test cases. The group's ID is unique in the context of this
385 test run. It's used elsewhere in the protocol to refer to this group without 392 test run. It's used elsewhere in the protocol to refer to this group without
386 including its full representation. 393 including its full representation.
387 394
388 The implicit group at the root of each test suite has `null` `name` and 395 The implicit group at the root of each test suite has `null` `name` and
389 `parentID` attributes. 396 `parentID` attributes.
390 397
391 The `line`, `column`, and `url` fields indicate the location the `group()` 398 The `line`, `column`, and `url` fields indicate the location the `group()`
392 function was called to create this group. They're treated as a unit: they'll 399 function was called to create this group. They're treated as a unit: they'll
393 either all be `null` or they'll all be non-`null`. The URL is always absolute, 400 either all be `null` or they'll all be non-`null`. The URL is always absolute,
394 and may be a `package:` URL. 401 and may be a `package:` URL.
395 402
396 ### Metadata 403 ### Metadata
397 404
398 ``` 405 ```
399 class Metadata { 406 class Metadata {
400 // Whether the test case will be skipped by the test runner.
401 bool skip; 407 bool skip;
402
403 // The reason the test case is skipped, if the user provided it.
404 String? skipReason; 408 String? skipReason;
405 } 409 }
406 ``` 410 ```
407 411
408 The metadata attached to a test by a user. 412 The metadata class is deprecated and should not be used.
409
410 Note that the `skip` field should not be considered authoritative. A test may be
411 skipped even if `skip` is set to `false`.
OLDNEW
« no previous file with comments | « no previous file | json_reporter.schema.json » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698