Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(105)

Side by Side Diff: common/api/dm/service/v1/graph_data.proto

Issue 1537883002: Initial distributor implementation (Closed) Base URL: https://chromium.googlesource.com/external/github.com/luci/luci-go@master
Patch Set: self review Created 4 years, 6 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
OLDNEW
1 // Copyright 2016 The LUCI Authors. All rights reserved. 1 // Copyright 2016 The LUCI Authors. All rights reserved.
2 // Use of this source code is governed under the Apache License, Version 2.0 2 // Use of this source code is governed under the Apache License, Version 2.0
3 // that can be found in the LICENSE file. 3 // that can be found in the LICENSE file.
4 4
5 syntax = "proto3"; 5 syntax = "proto3";
6 6
7 import "google/protobuf/timestamp.proto"; 7 import "google/protobuf/timestamp.proto";
8 import "google/protobuf/duration.proto";
8 9
9 import "github.com/luci/luci-go/common/api/template/template.proto"; 10 import "github.com/luci/luci-go/common/api/template/template.proto";
10 11
11 import "types.proto"; 12 import "types.proto";
12 13
13 package dm; 14 package dm;
14 15
16 message AbnormalFinish {
iannucci 2016/06/08 02:54:24 Here are all the abnormal finish status types. The
17 enum Status {
18 // This entity has a failed result.
19 //
20 // Executions: the distributor reported that the task executed and failed, O R
21 // the distributor reports success while the Execution is in the RUNNING
22 // state.
23 //
24 // Attempts: the last Execution had a FAILED Status.
25 //
26 // Retryable.
27 FAILED = 0;
iannucci 2016/06/08 02:54:24 I'm not super sure how to deal with this one: this
dnj (Google) 2016/06/09 18:00:57 I think things pre-recipe-engine can fail (e.g., b
iannucci 2016/06/15 00:46:02 Yeah I think this makes sense. For recipe quests w
28
29 // This entity failed in a bad way.
30 //
31 // Executions: The distributor told us that the job died violently while in
32 // the SCHEDULING, RUNNING or STOPPING state.
33 //
34 // Attempts: the last Execution had a CRASHED Status.
35 //
36 // Retryable.
37 CRASHED = 1;
38
39 // Waited too long for the job to start.
40 //
41 // Executions: the distributor couldn't start the job in time, OR DM failed
42 // to get a status update from the distributor in time (e.g. the state was
43 // SCHEDULING for too long).
44 //
45 // Attempts: the last Execution had an EXPIRED Status.
46 //
47 // Retryable.
48 EXPIRED = 2;
49
50 // The job started, but took too long.
51 //
52 // Executions: the distributor started the job, but it couldn't complete in
53 // time, OR DM failed to get a status update from the distributor in time
54 // (e.g. the state was RUNNING for too long).
55 //
56 // Attempts: the last Execution had an TIMED_OUT Status.
57 //
58 // Retryable.
59 TIMED_OUT = 3;
60
61 // The job was cancelled by an external entity (human, automated system).
62 //
63 // Executions: the distributor informing DM that the job was preemptively
64 // cancelled.
65 //
66 // Attempts: the last Execution had a CANCELLED Status, or this Attempt
67 // was cancelled via DM.
68 CANCELLED = 4;
iannucci 2016/06/08 02:54:24 later when DM supports cancellation directly, this
dnj (Google) 2016/06/09 18:00:57 nit: In American English, the verb cancel is usual
iannucci 2016/06/15 00:46:01 Who said protos were American? THEY LIVE IN CYBERS
dnj (Google) 2016/06/16 16:57:22 I'ma tell Obama.
69
70 // The job was prevented from running by the distributor (quota, permissions ,
71 // etc.)
72 //
73 // Executions: the distributor refused to run this job.
74 //
75 // Attempts: the last Execution had a REJECTED Status.
dnj (Google) 2016/06/09 18:00:57 Is this not retryable? If we're out of quota, we m
iannucci 2016/06/15 00:46:02 I think this should definitely be retried at a hig
76 REJECTED = 5;
77
78 // The job is unrecognized.
79 //
80 // Executions: the distributor doesn't know about this job, or has forgotten
81 // about it.
82 //
83 // Attempts: the last Execution had a REJECTED Status.
dnj (Google) 2016/06/09 18:00:57 MISSING Status?
iannucci 2016/06/15 00:46:01 oops
84 MISSING = 6;
85 }
86
87 Status status = 1;
88 string reason = 2;
89 }
90
15 message Quest { 91 message Quest {
16 message ID { 92 message ID {
17 string id = 1; 93 string id = 1;
18 } 94 }
19 ID id = 1; 95 ID id = 1;
20 96
21 // DNE is set to true if this Quest does not exist. None of the following 97 // DNE is set to true if this Quest does not exist. None of the following
22 // fields are valid if this is set to true. 98 // fields are valid if this is set to true.
23 bool DNE = 2; 99 bool DNE = 2;
24 100
25 message Desc { 101 message Desc {
102 // TODO(iannucci): have a 'simple_idempotent' quest mode which:
103 // * isn't allowed/expected to call any API methods (ActivateExecution,
104 // EnsureGraphData, or WalkGraph)
105 // * only provides data back through the distributor-specific 'state'
106 // field.
107 //
108 // Examples of use for this would be:
109 // * simple test binaries that run/output to an ISOLATED_OUTDIR
110 // * testing / ad-hoc bash scripts
111
26 string distributor_config_name = 1; 112 string distributor_config_name = 1;
27 string json_payload = 2; 113 string json_payload = 2;
114
115 message Meta {
116 // This names the user/service account for all Attempts on this quest. You
117 // must have permission to use this account when creating the Quest and/or
118 // Attempts.
119 string as_account = 1;
120
121 message Retry {
122 // The number of times in a row to retry Executions which have an
123 // ABNORMAL_FINISHED status of FAILED.
124 uint32 failed = 1;
125
126 // The number of times in a row to retry Executions which have an
127 // ABNORMAL_FINISHED status of EXPIRED.
128 uint32 expired = 2;
129
130 // The number of times in a row to retry Executions which have an
131 // ABNORMAL_FINISHED status of TIMED_OUT.
132 uint32 timed_out = 3;
133
134 // The number of times in a row to retry Executions which have an
135 // ABNORMAL_FINISHED status of CRASHED.
136 uint32 crashed = 4;
137 }
138
139 // This affects how DM will retry the job payload in various exceptional
140 // circumstances.
141 Retry retry = 2;
142 }
143
144 // This is metadata which doesn't affect the functionality of the payload,
145 // but does affect how DM and/or the distributor run/schedule that payload.
146 Meta meta = 3;
28 } 147 }
29 148
30 message TemplateSpec { 149 message TemplateSpec {
31 string project = 1; 150 string project = 1;
32 string ref = 2; 151 string ref = 2;
33 string version = 3; 152 string version = 3;
34 string name = 4; 153 string name = 4;
35 } 154 }
36 155
37 message Data { 156 message Data {
(...skipping 16 matching lines...) Expand all
54 string quest = 1; 173 string quest = 1;
55 uint32 id = 2; 174 uint32 id = 2;
56 } 175 }
57 ID id = 1; 176 ID id = 1;
58 177
59 // DNE is set to true if this Attempt does not exist. None of the following 178 // DNE is set to true if this Attempt does not exist. None of the following
60 // fields are valid if this is set to true. 179 // fields are valid if this is set to true.
61 bool DNE = 2; 180 bool DNE = 2;
62 181
63 enum State { 182 enum State {
64 NEEDS_EXECUTION = 0; 183 // The Attempt is waiting to be Executed
dnj (Google) 2016/06/09 18:00:57 nit: period at end.
iannucci 2016/06/15 00:46:02 Done.
184 SCHEDULING = 0;
185
186 // The Attempt is currently waiting for the current Execution to finish.
65 EXECUTING = 1; 187 EXECUTING = 1;
dnj (Google) 2016/06/09 18:00:57 IMO: s/the current/its current/
iannucci 2016/06/15 00:46:02 Done.
66 ADDING_DEPS = 2; 188
67 BLOCKED = 3; 189 // The Attempt is waiting for dependent Attempts to be resolved.
68 AWAITING_EXECUTION_STATE = 4; 190 WAITING = 2;
69 FINISHED = 5; 191
192 // The Attempt is in its final state.
193 FINISHED = 3;
194
195 // The Attempt is in an abnormal final state
dnj (Google) 2016/06/09 18:00:57 nit: period at end.
iannucci 2016/06/15 00:46:02 Done.
196 ABNORMAL_FINISHED = 4;
70 } 197 }
71 198
72 message Data { 199 message Data {
73 google.protobuf.Timestamp created = 1; 200 google.protobuf.Timestamp created = 1;
74 google.protobuf.Timestamp modified = 2; 201 google.protobuf.Timestamp modified = 2;
75 uint32 num_executions = 3; 202 uint32 num_executions = 3;
76 203
77 message NeedsExecution { 204 // This attempt is ready to be Executed, but hasn't been sent to the
78 google.protobuf.Timestamp pending = 1; 205 // distributor yet.
79 } 206 message Scheduling {}
80 207
208 // This attempt has a live Execution (with the specified ID). Check the
209 // Execution state for more information.
81 message Executing { 210 message Executing {
82 uint32 cur_execution_id = 1; 211 uint32 cur_execution_id = 1;
83 } 212 }
84 213
85 message AddingDeps { 214 // This attempt's last Execution stopped by adding dependencies.
86 uint32 num_adding = 1; 215 message Waiting {
87 uint32 num_waiting = 2;
88 }
89
90 message Blocked {
91 uint32 num_waiting = 1; 216 uint32 num_waiting = 1;
92 } 217 }
93 218
219 // This attempt is complete.
94 message Finished { 220 message Finished {
95 google.protobuf.Timestamp expiration = 1; 221 google.protobuf.Timestamp expiration = 1;
96 uint32 json_result_size = 2; 222 uint32 json_result_size = 2;
97 string json_result = 3; 223 string json_result = 3;
224
225 // This is the distributor-specific state of the final Execution.
226 string persistent_state_result = 4;
dnj (Google) 2016/06/09 18:00:57 Is "bytes" more appropriate? "string" implies UTF8
iannucci 2016/06/15 00:46:02 but string is the only map-key/immutable thing in
dnj (Google) 2016/06/16 16:57:22 Yeah I hear 'ya. But I think you did the right thi
98 } 227 }
99 228
100 oneof attempt_type { 229 oneof attempt_type {
101 NeedsExecution needs_execution = 4; 230 Scheduling scheduling = 5;
102 Executing executing = 5; 231 Executing executing = 6;
103 AddingDeps adding_deps = 6; 232 Waiting waiting = 7;
104 Blocked blocked = 7;
105 Finished finished = 8; 233 Finished finished = 8;
234 AbnormalFinish abnormal_finish = 9;
106 } 235 }
107 } 236 }
108 Data data = 3; 237 Data data = 3;
109 238
110 // key is the `id` field of the Execution.ID 239 // key is the `id` field of the Execution.ID
111 map<uint32, Execution> executions = 4; 240 map<uint32, Execution> executions = 4;
112 241
113 dm.AttemptList fwd_deps = 5; 242 dm.AttemptList fwd_deps = 5;
114 dm.AttemptList back_deps = 6; 243 dm.AttemptList back_deps = 6;
115 244
(...skipping 47 matching lines...) Expand 10 before | Expand all | Expand 10 after
163 } 292 }
164 293
165 message ID { 294 message ID {
166 string quest = 1; 295 string quest = 1;
167 uint32 attempt = 2; 296 uint32 attempt = 2;
168 uint32 id = 3; 297 uint32 id = 3;
169 } 298 }
170 ID id = 1; 299 ID id = 1;
171 300
172 enum State { 301 enum State {
173 // The execution has been accepted by the distributor, but is not running ye t 302 // The execution has been accepted by the distributor, but is not running
174 SCHEDULED = 0; 303 // yet.
304 SCHEDULING = 0;
175 305
176 // The execution is running 306 // The execution is running (has activated with DM).
177 RUNNING = 1; 307 RUNNING = 1;
178 308
179 // The execution was unable to be accepted by the distributor 309 // The execution has been told to stop by DM, but we haven't heard from
180 REJECTED = 2; 310 // the distributor yet.
311 STOPPING = 2;
181 312
182 // The execution was accepted by the distributor, but couldn't run in time. 313 // The execution is in its final state.
183 TIMED_OUT = 3; 314 FINISHED = 3;
184 315
185 // The execution ran and completed 316 // The execution is in an abnormal final state
186 FINISHED = 4; 317 ABNORMAL_FINISHED = 4;
187
188 // The execution ran, but the distributor claims it did not complete
189 FAILED = 5;
190
191 // The distributor claims to not know anything about this execution
192 MISSING = 6;
193
194 // Some entity (DM, Human, Distributor) requested that this execution not ru n.
195 CANCELLED = 7;
196 } 318 }
197 319
198 message Data { 320 message Data {
199 State state = 1; 321 google.protobuf.Timestamp created = 1;
200 string state_reason = 2; 322 google.protobuf.Timestamp modified = 2;
201 323
202 google.protobuf.Timestamp created = 3; 324 message DistributorInfo {
325 string config_name = 1;
326 string config_version = 2;
327 string token = 3;
328 string url = 4;
329 }
330 DistributorInfo distributor_info = 3;
203 331
204 string distributor_token = 4; 332 message Scheduling {}
205 string distributor_info_url = 5; 333
334 message Running {}
335
336 message Stopping {}
337
338 message Finished {
339 string persistent_state = 1;
340 }
341
342 oneof execution_type {
343 Scheduling scheduling = 4;
344 Running running = 5;
345 Stopping stopping = 6;
346 Finished finished = 7;
347 AbnormalFinish abnormal_finish = 8;
348 }
206 } 349 }
207 Data data = 2; 350 Data data = 2;
208 351
209 // Partial is true iff the request asked for Executions, but wasn't able to 352 // Partial is true iff the request asked for Executions, but wasn't able to
210 // completely fill them. 353 // completely fill them.
211 bool partial = 16; 354 bool partial = 16;
212 } 355 }
213 356
214 // GraphData defines all of the DM graph data that may be returned from DM. 357 // GraphData defines all of the DM graph data that may be returned from DM.
215 // 358 //
(...skipping 22 matching lines...) Expand all
238 // * max response size limit 381 // * max response size limit
239 // * max time limit (e.g. WalkGraphReq.MaxTime) being hit 382 // * max time limit (e.g. WalkGraphReq.MaxTime) being hit
240 // * non-terminal errors encountered during the request (HadErrors will also 383 // * non-terminal errors encountered during the request (HadErrors will also
241 // be true in this case). 384 // be true in this case).
242 // 385 //
243 // Note that this is different than the Partial booleans: This refers 386 // Note that this is different than the Partial booleans: This refers
244 // specifically to situations when Queries do not run to completion. 387 // specifically to situations when Queries do not run to completion.
245 bool had_more = 3; 388 bool had_more = 3;
246 } 389 }
247 390
OLDNEW

Powered by Google App Engine
This is Rietveld 408576698