Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(467)

Side by Side Diff: json_reporter.md

Issue 1461293005: Add a JSON reporter. (Closed) Base URL: git@github.com:dart-lang/test@master
Patch Set: Created 5 years ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | json_reporter.schema.json » ('j') | json_reporter.schema.json » ('J')
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
(Empty)
1 JSON Reporter Protocol
2 ======================
3
4 The test runner supports a JSON reporter which provides a machine-readable
5 representation of the test runner's progress. This reporter is intended for use
6 by IDEs and other tools to present a custom view of the test runner's operation
7 without needing to parse output intended for humans.
8
9 ## Usage
10
11 Pass the `--reporter json` command-line flag to the test runner to activate the
12 JSON reporter.
13
14 pub run test --reporter json <path-to-test-file>
15
16 The JSON stream will be emitted via standard output. It will be a stream of JSON
17 objects, separated by newlines.
18
19 See `json_reporter.schema.json` for a formal description of the protocol schema.
20 See `test/runner/json_reporter_test.dart` for some sample output.
21
22 ## Compatibility
23
24 The protocol emitted by the JSON reporter is considered part of the public API
25 of the `test` package, and is subject to its [semantic versioning][semver]
26 restrictions. In particular:
27
28 [semver]: https://www.dartlang.org/tools/pub/versioning.html#semantic-versions
29
30 * No new feature will be added to the protocol without increasing the test
31 package's minor version number.
32
33 * No breaking change will be made to the protocol without increasing the test
34 package's major version number.
35
36 The following changes are not considered breaking. This is not necessarily a
37 comprehensive list.
38
39 * Adding a new attribute to an existing object.
40
41 * Adding a new type of any object with a `type` parameter.
42
43 * Adding new test state values.
44
45 ## Reading this Document
46
47 Each major type of JSON object used by the protocol is described by a class.
48 Classes have names which are referred to in this document, but are not used as
49 part of the protocol. Classes have typed attributes, which refer to the types
50 and names of attributes in the JSON objects. If an attribute's type is another
51 class, that refers to a nested object. The special type `List<...>` indicates a
52 JSON list of the given type.
53
54 Classes can "extend" one another, meaning that the subclass has all the
55 properties of the superclass. Concrete subclasses can be distinguished by the
56 specific value of their `type` attribute. Classes may be abstract, indicating
57 that only their subclasses will ever be used.
58
59 ## Events
60
61 ### Event
62
63 ```
64 abstract class Event {
65 // The type of the event.
66 //
67 // This is always one of the subclass types listed below.
68 String type;
69
70 // The time (in milliseconds) that has elapsed since the test runner started.
71 int time;
72 }
73 ```
74
75 This is the root class of the protocol. All root-level objects emitted by the
76 JSON reporter will be subclasses of `Event`.
77
78 ### StartEvent
79
80 ```
81 class StartEvent extends Event {
82 String type = "start";
83
84 // The version of the JSON reporter protocol being used.
85 //
86 // This is a semantic version, but it reflects only the version of the
87 // protocol—it's not identical to the version of the test runner itself.
88 String protocolVersion;
89
90 // The version of the test runner being used.
91 String runnerVersion;
92 }
93 ```
94
95 A single start event is emitted before any other events. It indicates that the
96 test runner has started running.
97
98 ### TestStartEvent
99
100 ```
101 class TestStartEvent extends Event {
102 String type = "testStart";
103
104 // Metadata about the test that started.
105 Test test;
106 }
107 ```
108
109 An event emitted when a test begins running. This is the only event that
110 contains the full metadata about a test; future events will refer to the test by
111 its opaque ID.
112
113 The test should be considered skipped if `test.metadata.skip` is `true`.
114
115 ### PrintEvent
116
117 ```
118 class PrintEvent extends Event {
119 String type = "print";
120
121 // The ID of the test that printed a message.
122 int testID;
123
124 // The message that was printed.
125 String message;
126 }
127 ```
128
129 A `PrintEvent` indicates that a test called `print()` and wishes to display
130 output.
131
132 ### ErrorEvent
133
134 ```
135 class ErrorEvent extends Event {
136 String type = "error";
137
138 // The ID of the test that experienced the error.
139 int testID;
140
141 // The result of calling toString() on the error object.
142 String error;
143
144 // The error's stack trace, in the stack_trace package format.
145 String stackTrace;
146
147 // Whether the error was a TestFailure.
148 bool isFailure;
149 }
150 ```
151
152 A `ErrorEvent` indicates that a test encountered an uncaught error. Note
153 that this may happen even after the test has completed, in which case it should
154 be considered to have failed.
155
156 If a test is asynchronous, it may encounter multiple errors, which will result
157 in multiple `ErrorEvent`s.
158
159 ### TestDoneEvent
160
161 ```
162 class TestDoneEvent extends Event {
163 String type = "testDone";
164
165 // The ID of the test that completed.
166 int testID;
167
168 // The result of the test.
169 String result;
170
171 // Whether the test's result should be hidden.
172 bool hidden;
173 }
174 ```
175
176 An event emitted when a test completes. The `result` attribute indicates the
177 result of the test:
178
179 * `"success"` if the test had no errors.
180
181 * `"failure"` if the test had a `TestFailure` but no other errors.
182
183 * `"error"` if the test had an error other than a `TestFailure`.
184
185 If the test encountered an error, the `TestDoneEvent` will be emitted after the
186 corresponding `ErrorEvent`.
187
188 The `hidden` attribute indicates that the test's result should be hidden and not
189 counted towards the total number of tests run for the suite. This is true for
190 virtual tests created for loading test suites, `setUpAll()`, and
191 `tearDownAll()`. Only successful tests will be hidden.
192
193 Note that it's possible for a test to encounter an error after completing. In
194 that case, it should be considered to have failed, but no additional
195 `TestDoneEvent` will be emitted. If a previously-hidden test encounters an
196 error after completing, it should be made visible.
197
198 ### DoneEvent
199
200 ```
201 class DoneEvent extends Event {
202 String type = "done";
203
204 // Whether all tests succeeded (or were skipped).
205 bool success;
206 }
207 ```
208
209 An event indicating the result of the entire test run. This will be the final
210 event emitted by the reporter.
211
212 ## Test Information
213
214 ### Test
215
216 ```
217 class Test {
218 // An opaque ID for the test.
219 int id;
220
221 // The name of the test, including prefixes from any containing groups.
222 String name;
223
224 // The test's metadata, including metadata from any containing groups and the
225 // test suite itself.
226 Metadata metadata;
227 }
228 ```
229
230 A single test case. The test's ID is unique in the context of this test run.
231 It's used elsewhere in the protocol to refer to this test without including its
232 full representation.
233
234 ### Metadata
235
236 ```
237 class Metadata {
238 // Whether the test case will be skipped by the test runner.
239 bool skip;
240
241 // The reason the test case is skipped, if the user provided it.
242 String? skipReason;
243 }
244 ```
245
246 The metadata attached to a test by a user.
OLDNEW
« no previous file with comments | « no previous file | json_reporter.schema.json » ('j') | json_reporter.schema.json » ('J')

Powered by Google App Engine
This is Rietveld 408576698