tools/android/loading Move task creation to background and add dashboard

tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of jobs is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.

Committed: https://crrev.com/4953d75f1ebef626ca9dfcc20630193dedcfb8ea
Cr-Commit-Position: refs/heads/master@{#399175}

[droger, 2016-06-08 14:07:43]

Patchset #3 (id:40001) has been deleted

[droger, 2016-06-08 14:07:48]

Patchset #2 (id:20001) has been deleted

[droger, 2016-06-08 14:07:53]

Patchset #1 (id:1) has been deleted

[droger, 2016-06-08 15:19:28]

Patchset #1 (id:60001) has been deleted

[droger, 2016-06-08 15:19:34]

Patchset #1 (id:80001) has been deleted

[droger, 2016-06-08 15:19:39]

Patchset #1 (id:100001) has been deleted

[droger, 2016-06-08 15:19:45]

Patchset #1 (id:120001) has been deleted

[droger, 2016-06-08 15:25:32]

Description was changed from

==========
tools/android/loading Move task creation to background and add dashboard
==========

to

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of tasks is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once.
==========

[droger, 2016-06-08 15:25:51]

Description was changed from

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of tasks is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once.
==========

to

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of tasks is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.
==========

[droger, 2016-06-08 15:26:14]

droger@chromium.org changed reviewers:
+ mattcary@chromium.org

[droger, 2016-06-08 15:26:35]

Description was changed from

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of tasks is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.
==========

to

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of jobs is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.
==========

[mattcary, 2016-06-09 15:40:06]

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
File tools/android/loading/cloud/frontend/clovis_frontend.py (right):

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/clovis_frontend.py:363: pass  # Catch all
exceptions so that AppEngine does not retry indefinitely.
Log the exception so if tasks disappear we have a chance at figuring out why?

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/clovis_frontend.py:392: frontend_job.status
= 'building_task_url'
Do you need to put these status update values, or will it magically update?

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
File tools/android/loading/cloud/frontend/frontend_job.py (right):

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/frontend_job.py:32: @staticmethod
We prefer @classmethod rather than @staticmethod. That would enable subclasses
to work as epxected (eg, instead of FrontendJob._GetParentKeyFromTag one could
do cls._GetParentKeyFromTag and things would override as expected).

[droger, 2016-06-10 12:14:04]

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
File tools/android/loading/cloud/frontend/clovis_frontend.py (right):

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/clovis_frontend.py:363: pass  # Catch all
exceptions so that AppEngine does not retry indefinitely.
On 2016/06/09 15:40:06, mattcary wrote:
> Log the exception so if tasks disappear we have a chance at figuring out why?

Done.

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/clovis_frontend.py:392: frontend_job.status
= 'building_task_url'
On 2016/06/09 15:40:06, mattcary wrote:
> Do you need to put these status update values, or will it magically update?

You need to put.

This is only done at the end (line 368) and not every time the status changes.

I'm adding a couple more calls now, but not every time the status changes
because I'm worried that calling it too often could cause problems.

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
File tools/android/loading/cloud/frontend/frontend_job.py (right):

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/frontend_job.py:32: @staticmethod
On 2016/06/09 15:40:06, mattcary wrote:
> We prefer @classmethod rather than @staticmethod. That would enable subclasses
> to work as epxected (eg, instead of FrontendJob._GetParentKeyFromTag one could
> do cls._GetParentKeyFromTag and things would override as expected).

Done.

[mattcary, 2016-06-10 12:17:27]

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
File tools/android/loading/cloud/frontend/clovis_frontend.py (right):

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/clovis_frontend.py:392: frontend_job.status
= 'building_task_url'
On 2016/06/10 12:14:04, droger wrote:
> On 2016/06/09 15:40:06, mattcary wrote:
> > Do you need to put these status update values, or will it magically update?
> 
> You need to put.
> 
> This is only done at the end (line 368) and not every time the status changes.
> 
> I'm adding a couple more calls now, but not every time the status changes
> because I'm worried that calling it too often could cause problems.

Shouldn't we put after each return, so we have the current status?

This is only going to be executed from the push queue, and so is going to have <
10 queries/minute. I shouldn't think that would be a problem for updates in the
datastore.

[droger, 2016-06-10 12:58:27]

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
File tools/android/loading/cloud/frontend/clovis_frontend.py (right):

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/clovis_frontend.py:392: frontend_job.status
= 'building_task_url'
On 2016/06/10 12:17:27, mattcary wrote:
> On 2016/06/10 12:14:04, droger wrote:
> > On 2016/06/09 15:40:06, mattcary wrote:
> > > Do you need to put these status update values, or will it magically
update?
> > 
> > You need to put.
> > 
> > This is only done at the end (line 368) and not every time the status
changes.
> > 
> > I'm adding a couple more calls now, but not every time the status changes
> > because I'm worried that calling it too often could cause problems.
> 
> Shouldn't we put after each return, so we have the current status?
> 
> This is only going to be executed from the push queue, and so is going to have
<
> 10 queries/minute. I shouldn't think that would be a problem for updates in
the
> datastore.

put() will be executed after each return, by the caller code:
this function is called on line 361, and put() is called line 368.

[mattcary, 2016-06-10 13:24:20]

lgtm

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
File tools/android/loading/cloud/frontend/clovis_frontend.py (right):

https://codereview.chromium.org/2041193008/diff/180001/tools/android/loading/...
tools/android/loading/cloud/frontend/clovis_frontend.py:392: frontend_job.status
= 'building_task_url'
On 2016/06/10 12:58:27, droger wrote:
> On 2016/06/10 12:17:27, mattcary wrote:
> > On 2016/06/10 12:14:04, droger wrote:
> > > On 2016/06/09 15:40:06, mattcary wrote:
> > > > Do you need to put these status update values, or will it magically
> update?
> > > 
> > > You need to put.
> > > 
> > > This is only done at the end (line 368) and not every time the status
> changes.
> > > 
> > > I'm adding a couple more calls now, but not every time the status changes
> > > because I'm worried that calling it too often could cause problems.
> > 
> > Shouldn't we put after each return, so we have the current status?
> > 
> > This is only going to be executed from the push queue, and so is going to
have
> <
> > 10 queries/minute. I shouldn't think that would be a problem for updates in
> the
> > datastore.
> 
> put() will be executed after each return, by the caller code:
> this function is called on line 361, and put() is called line 368.

Ah! I was just looking in SpawnTasks. Thanks.

[droger, 2016-06-10 13:42:13]

The CQ bit was checked by droger@chromium.org

[commit-bot: I haz the power, 2016-06-10 13:42:30]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/2041193008/200001

[commit-bot: I haz the power, 2016-06-10 14:15:55]

Description was changed from

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of jobs is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.
==========

to

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of jobs is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.
==========

[commit-bot: I haz the power, 2016-06-10 14:15:56]

Committed patchset #4 (id:200001)

[commit-bot: I haz the power, 2016-06-10 14:15:58]

CQ bit was unchecked

[commit-bot: I haz the power, 2016-06-10 14:18:00]

Description was changed from

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of jobs is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.
==========

to

==========
tools/android/loading Move task creation to background and add dashboard

This CL moves the tasks creation to the background queue,
to avoid AppEngine timeouts.

It also adds a way to list the current jobs and track them.

The list of jobs is persisted in the datastore.

There is more work to be done in follow up CLs:
- improving the UI for the job dashboard
- possibly replacing the per-job polling by a global
  polling that would poll all active jobs at once, in order
  to simplify the code.

Committed: https://crrev.com/4953d75f1ebef626ca9dfcc20630193dedcfb8ea
Cr-Commit-Position: refs/heads/master@{#399175}
==========

[commit-bot: I haz the power, 2016-06-10 14:18:02]

Patchset 4 (id:??) landed as
https://crrev.com/4953d75f1ebef626ca9dfcc20630193dedcfb8ea
Cr-Commit-Position: refs/heads/master@{#399175}