Create tryserver appengine app.

Create tryserver appengine app.

This appengine app is a step towards decentralized job control. It accepts POSTs
of json blobs describing buildbot jobs to its 'push' endpoint. Then, requests to
the 'pull' endpoint will be served not-yet-picked-up jobs. The poller can then
POST to the 'accept' endpoint to indicate that it has successfully picked up and
started a job. The 'peek' endpoint serves to inspect the job queue without
marking any jobs as potentially taken.

This is intended to interact with 'git try' on the client side to push blobs to
the app, and a new buildbot poller and scheduler to consume blobs and start
jobs.

R=cmp@google.com

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=219250

[agable, 2013-08-21 13:43:06]

PTAL. Now in the new-and-improved location. cmp@ on review because he started
taking a look at the first iteration of this CL, xusydoc@ because iannucci@ is
OOO and cmp@ is always busy :)

[cmp, 2013-08-22 04:44:12]

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py
File chromium-jobqueue/app.py (right):

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:1: # Copyright (c) 2012 The Chromium Authors. All
rights reserved.
2013?

There should be a LICENSE file in the CL, too.

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:23: tryserver. Default is epoch 0, so new jobs are
"old".
tryserver -> other name

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:25: a trysever and can be dropped.
tryserver -> other name

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:94: time_threshold = datetime.datetime.utcnow() -
datetime.timedelta(seconds=30)
wrap at 80 chars

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:119: job.taken = True
I think this would be a perfect place to use pull queues:
https://developers.google.com/appengine/docs/python/taskqueue/overview-pull

The lease period could allow us to ensure the job is actively being worked on by
a slave.  The payload could be the JSON data.  If the slave halts or goes AWOL,
the task would be placed back into the queue for another slave to pick up.

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.yaml
File chromium-jobqueue/app.yaml (right):

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.yaml#ne...
chromium-jobqueue/app.yaml:1: application: chromium-tryserver
chromium-jobqueue?

[agable, 2013-08-22 13:43:42]

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py
File chromium-jobqueue/app.py (right):

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:1: # Copyright (c) 2012 The Chromium Authors. All
rights reserved.
On 2013/08/22 04:44:12, cmp wrote:
> 2013?
> 
> There should be a LICENSE file in the CL, too.

Done, copied from depot_tools. Out of curiosity, why don't build,
chromium-build, etc. have LICENSE files? Just because we've forgotten them?

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:23: tryserver. Default is epoch 0, so new jobs are
"old".
On 2013/08/22 04:44:12, cmp wrote:
> tryserver -> other name

Done.

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:25: a trysever and can be dropped.
On 2013/08/22 04:44:12, cmp wrote:
> tryserver -> other name

Done.

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:94: time_threshold = datetime.datetime.utcnow() -
datetime.timedelta(seconds=30)
On 2013/08/22 04:44:12, cmp wrote:
> wrap at 80 chars

Done.

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.py#newc...
chromium-jobqueue/app.py:119: job.taken = True
On 2013/08/22 04:44:12, cmp wrote:
> I think this would be a perfect place to use pull queues:
> https://developers.google.com/appengine/docs/python/taskqueue/overview-pull
> 
> The lease period could allow us to ensure the job is actively being worked on
by
> a slave.  The payload could be the JSON data.  If the slave halts or goes
AWOL,
> the task would be placed back into the queue for another slave to pick up.

Agreed. Right now, the Pull/Accept paradigm is just best-effort -- this app is
not making sure the job succeeds, it's just making sure the job gets picked up
by a master. As this gets closer to MCP and jobs are being picked up directly by
slaves, it will have true retry logic built in, at which point using pull queues
will be perfect. For now, though, I think pull/accept + the JsonPoller is
simpler than lease/delete + a new TaskQueuePoller using the full Google API
Client Library.

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.yaml
File chromium-jobqueue/app.yaml (right):

https://codereview.chromium.org/23093011/diff/1/chromium-jobqueue/app.yaml#ne...
chromium-jobqueue/app.yaml:1: application: chromium-tryserver
On 2013/08/22 04:44:12, cmp wrote:
> chromium-jobqueue?

Done.

[cmp_google, 2013-08-23 03:10:45]

lgtm

[agable, 2013-08-23 07:43:35]

Committed patchset #2 manually as r219250 (presubmit successful).