Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(242)

Side by Side Diff: doc/json_reporter.md

Issue 2099503002: Add a dedicated Result for skipped tests. (Closed) Base URL: git@github.com:dart-lang/test@master
Patch Set: Code review changes Created 4 years, 5 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | json_reporter.schema.json » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 JSON Reporter Protocol 1 JSON Reporter Protocol
2 ====================== 2 ======================
3 3
4 The test runner supports a JSON reporter which provides a machine-readable 4 The test runner supports a JSON reporter which provides a machine-readable
5 representation of the test runner's progress. This reporter is intended for use 5 representation of the test runner's progress. This reporter is intended for use
6 by IDEs and other tools to present a custom view of the test runner's operation 6 by IDEs and other tools to present a custom view of the test runner's operation
7 without needing to parse output intended for humans. 7 without needing to parse output intended for humans.
8 8
9 Note that the test runner is highly asynchronous, and users of this protocol 9 Note that the test runner is highly asynchronous, and users of this protocol
10 shouldn't make assumptions about the ordering of events beyond what's explicitly 10 shouldn't make assumptions about the ordering of events beyond what's explicitly
(...skipping 133 matching lines...) Expand 10 before | Expand all | Expand 10 after
144 ``` 144 ```
145 145
146 A group event is emitted before any `TestStartEvent`s for tests in a given 146 A group event is emitted before any `TestStartEvent`s for tests in a given
147 group. This is the only event that contains the full metadata about a group; 147 group. This is the only event that contains the full metadata about a group;
148 future events will refer to the group by its opaque ID. 148 future events will refer to the group by its opaque ID.
149 149
150 This includes the implicit group at the root of each suite, which has a `null` 150 This includes the implicit group at the root of each suite, which has a `null`
151 name. However, it does *not* include implicit groups for the virtual suites 151 name. However, it does *not* include implicit groups for the virtual suites
152 generated to represent loading test files. 152 generated to represent loading test files.
153 153
154 The group should be considered skipped if `group.metadata.skip` is `true`. When 154 If the group is skipped, a single `TestStartEvent` will be emitted for a test
155 a group is skipped, a single `TestStartEvent` will be emitted for a test within 155 within the group, followed by a `TestDoneEvent` marked as skipped. The
156 that group that will also be skipped. 156 `group.metadata.skip` field should *not* be considered authoritative for
157 determining whether a group is skipped.
157 158
158 ### TestStartEvent 159 ### TestStartEvent
159 160
160 ``` 161 ```
161 class TestStartEvent extends Event { 162 class TestStartEvent extends Event {
162 String type = "testStart"; 163 String type = "testStart";
163 164
164 // Metadata about the test that started. 165 // Metadata about the test that started.
165 Test test; 166 Test test;
166 } 167 }
167 ``` 168 ```
168 169
169 An event emitted when a test begins running. This is the only event that 170 An event emitted when a test begins running. This is the only event that
170 contains the full metadata about a test; future events will refer to the test by 171 contains the full metadata about a test; future events will refer to the test by
171 its opaque ID. 172 its opaque ID.
172 173
173 The test should be considered skipped if `test.metadata.skip` is `true`. 174 If the test is skipped, its `TestDoneEvent` will have `skipped` set to `true`.
175 The `test.metadata.skip` field should *not* be considered authoritative for
176 determining whether a test is skipped.
174 177
175 ### PrintEvent 178 ### PrintEvent
176 179
177 ``` 180 ```
178 class PrintEvent extends Event { 181 class PrintEvent extends Event {
179 String type = "print"; 182 String type = "print";
180 183
181 // The ID of the test that printed a message. 184 // The ID of the test that printed a message.
182 int testID; 185 int testID;
183 186
(...skipping 39 matching lines...) Expand 10 before | Expand all | Expand 10 after
223 String type = "testDone"; 226 String type = "testDone";
224 227
225 // The ID of the test that completed. 228 // The ID of the test that completed.
226 int testID; 229 int testID;
227 230
228 // The result of the test. 231 // The result of the test.
229 String result; 232 String result;
230 233
231 // Whether the test's result should be hidden. 234 // Whether the test's result should be hidden.
232 bool hidden; 235 bool hidden;
236
237 // Whether the test (or some part of it) was skipped.
238 bool skipped;
233 } 239 }
234 ``` 240 ```
235 241
236 An event emitted when a test completes. The `result` attribute indicates the 242 An event emitted when a test completes. The `result` attribute indicates the
237 result of the test: 243 result of the test:
238 244
239 * `"success"` if the test had no errors. 245 * `"success"` if the test had no errors.
240 246
241 * `"failure"` if the test had a `TestFailure` but no other errors. 247 * `"failure"` if the test had a `TestFailure` but no other errors.
242 248
(...skipping 150 matching lines...) Expand 10 before | Expand all | Expand 10 after
393 class Metadata { 399 class Metadata {
394 // Whether the test case will be skipped by the test runner. 400 // Whether the test case will be skipped by the test runner.
395 bool skip; 401 bool skip;
396 402
397 // The reason the test case is skipped, if the user provided it. 403 // The reason the test case is skipped, if the user provided it.
398 String? skipReason; 404 String? skipReason;
399 } 405 }
400 ``` 406 ```
401 407
402 The metadata attached to a test by a user. 408 The metadata attached to a test by a user.
409
410 Note that the `skip` field should not be considered authoritative. A test may be
411 skipped even if `skip` is set to `false`.
OLDNEW
« no previous file with comments | « no previous file | json_reporter.schema.json » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698