OLD | NEW |
1 JSON Reporter Protocol | 1 JSON Reporter Protocol |
2 ====================== | 2 ====================== |
3 | 3 |
4 The test runner supports a JSON reporter which provides a machine-readable | 4 The test runner supports a JSON reporter which provides a machine-readable |
5 representation of the test runner's progress. This reporter is intended for use | 5 representation of the test runner's progress. This reporter is intended for use |
6 by IDEs and other tools to present a custom view of the test runner's operation | 6 by IDEs and other tools to present a custom view of the test runner's operation |
7 without needing to parse output intended for humans. | 7 without needing to parse output intended for humans. |
8 | 8 |
9 Note that the test runner is highly asynchronous, and users of this protocol | 9 Note that the test runner is highly asynchronous, and users of this protocol |
10 shouldn't make assumptions about the ordering of events beyond what's explicitly | 10 shouldn't make assumptions about the ordering of events beyond what's explicitly |
(...skipping 133 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
144 ``` | 144 ``` |
145 | 145 |
146 A group event is emitted before any `TestStartEvent`s for tests in a given | 146 A group event is emitted before any `TestStartEvent`s for tests in a given |
147 group. This is the only event that contains the full metadata about a group; | 147 group. This is the only event that contains the full metadata about a group; |
148 future events will refer to the group by its opaque ID. | 148 future events will refer to the group by its opaque ID. |
149 | 149 |
150 This includes the implicit group at the root of each suite, which has a `null` | 150 This includes the implicit group at the root of each suite, which has a `null` |
151 name. However, it does *not* include implicit groups for the virtual suites | 151 name. However, it does *not* include implicit groups for the virtual suites |
152 generated to represent loading test files. | 152 generated to represent loading test files. |
153 | 153 |
154 The group should be considered skipped if `group.metadata.skip` is `true`. When | 154 If the group is skipped, a single `TestStartEvent` will be emitted for a test |
155 a group is skipped, a single `TestStartEvent` will be emitted for a test within | 155 within the group, followed by a `TestDoneEvent` marked as skipped. The |
156 that group that will also be skipped. | 156 `group.metadata.skip` field should *not* be considered authoritative for |
| 157 determining whether a group is skipped. |
157 | 158 |
158 ### TestStartEvent | 159 ### TestStartEvent |
159 | 160 |
160 ``` | 161 ``` |
161 class TestStartEvent extends Event { | 162 class TestStartEvent extends Event { |
162 String type = "testStart"; | 163 String type = "testStart"; |
163 | 164 |
164 // Metadata about the test that started. | 165 // Metadata about the test that started. |
165 Test test; | 166 Test test; |
166 } | 167 } |
167 ``` | 168 ``` |
168 | 169 |
169 An event emitted when a test begins running. This is the only event that | 170 An event emitted when a test begins running. This is the only event that |
170 contains the full metadata about a test; future events will refer to the test by | 171 contains the full metadata about a test; future events will refer to the test by |
171 its opaque ID. | 172 its opaque ID. |
172 | 173 |
173 The test should be considered skipped if `test.metadata.skip` is `true`. | 174 If the test is skipped, its `TestDoneEvent` will have `skipped` set to `true`. |
| 175 The `test.metadata.skip` field should *not* be considered authoritative for |
| 176 determining whether a test is skipped. |
174 | 177 |
175 ### PrintEvent | 178 ### PrintEvent |
176 | 179 |
177 ``` | 180 ``` |
178 class PrintEvent extends Event { | 181 class PrintEvent extends Event { |
179 String type = "print"; | 182 String type = "print"; |
180 | 183 |
181 // The ID of the test that printed a message. | 184 // The ID of the test that printed a message. |
182 int testID; | 185 int testID; |
183 | 186 |
(...skipping 39 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
223 String type = "testDone"; | 226 String type = "testDone"; |
224 | 227 |
225 // The ID of the test that completed. | 228 // The ID of the test that completed. |
226 int testID; | 229 int testID; |
227 | 230 |
228 // The result of the test. | 231 // The result of the test. |
229 String result; | 232 String result; |
230 | 233 |
231 // Whether the test's result should be hidden. | 234 // Whether the test's result should be hidden. |
232 bool hidden; | 235 bool hidden; |
| 236 |
| 237 // Whether the test (or some part of it) was skipped. |
| 238 bool skipped; |
233 } | 239 } |
234 ``` | 240 ``` |
235 | 241 |
236 An event emitted when a test completes. The `result` attribute indicates the | 242 An event emitted when a test completes. The `result` attribute indicates the |
237 result of the test: | 243 result of the test: |
238 | 244 |
239 * `"success"` if the test had no errors. | 245 * `"success"` if the test had no errors. |
240 | 246 |
241 * `"failure"` if the test had a `TestFailure` but no other errors. | 247 * `"failure"` if the test had a `TestFailure` but no other errors. |
242 | 248 |
(...skipping 150 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
393 class Metadata { | 399 class Metadata { |
394 // Whether the test case will be skipped by the test runner. | 400 // Whether the test case will be skipped by the test runner. |
395 bool skip; | 401 bool skip; |
396 | 402 |
397 // The reason the test case is skipped, if the user provided it. | 403 // The reason the test case is skipped, if the user provided it. |
398 String? skipReason; | 404 String? skipReason; |
399 } | 405 } |
400 ``` | 406 ``` |
401 | 407 |
402 The metadata attached to a test by a user. | 408 The metadata attached to a test by a user. |
| 409 |
| 410 Note that the `skip` field should not be considered authoritative. A test may be |
| 411 skipped even if `skip` is set to `false`. |
OLD | NEW |