LogDog: Support per-namespace expired archival.

LogDog: Support per-namespace expired archival.

Currently, expired archival only operates on the root namesapce.

Change archival to use named delayed Tumble tasks, scheduling the
cleanup archival on stream registration and clearing it on stream
termination.

BUG=

Committed: https://github.com/luci/luci-go/commit/4be9524ee752a3d4740a26c0debc1807a2e68f7e

[dnj, 2016-04-25 16:24:59]

dnj@chromium.org changed reviewers:
+ hinoka@chromium.org, iannucci@chromium.org

[dnj, 2016-04-25 16:25:00]

PTAL

[Ryan Tseng, 2016-04-28 20:16:02]

hinoka@google.com changed reviewers:
+ hinoka@google.com

[Ryan Tseng, 2016-04-28 20:16:04]

lgtm

https://codereview.chromium.org/1910633006/diff/20001/appengine/logdog/coordi...
File appengine/logdog/coordinator/backend/archiveCron.go (right):

https://codereview.chromium.org/1910633006/diff/20001/appengine/logdog/coordi...
appengine/logdog/coordinator/backend/archiveCron.go:79:
log.WithError(err).Errorf(c, "Failed to add non-namesapced scan task.")
namespaced

https://codereview.chromium.org/1910633006/diff/20001/appengine/logdog/coordi...
appengine/logdog/coordinator/backend/archiveCron.go:92: ic := c
Maybe add small comment about why the context is being copied

https://codereview.chromium.org/1910633006/diff/20001/appengine/logdog/coordi...
appengine/logdog/coordinator/backend/archiveCron.go:113: return
errors.New("failed to create all scan tasks")
All? Or just some?

https://codereview.chromium.org/1910633006/diff/20001/appengine/logdog/coordi...
appengine/logdog/coordinator/backend/archiveCron.go:168: //
?

https://codereview.chromium.org/1910633006/diff/20001/appengine/logdog/coordi...
File appengine/logdog/coordinator/backend/backend.go (right):

https://codereview.chromium.org/1910633006/diff/20001/appengine/logdog/coordi...
appengine/logdog/coordinator/backend/backend.go:36: r.POST("/archive/cron/scan",
h(gaemiddleware.RequireTaskQueue("", b.HandleArchiveScanTask)))
nit: this could be less than 80

[dnj, 2016-04-29 02:44:56]

Description was changed from

==========
LogDog: Support per-namespace expired archival.

Currently, expired archival only operates on the root namesapce. Expand
this so that the initial cron dispatches one scanning/handling task per
project namespace using a push queue.

BUG=
==========

to

==========
LogDog: Support per-namespace expired archival.

Currently, expired archival only operates on the root namesapce.

Change archival to use named delayed Tumble tasks, scheduling the
cleanup archival on stream registration and clearing it on stream
termination.

BUG=
==========

[dnj, 2016-04-29 02:53:35]

iannucci@, after discussion I switched this to use delayed mutations. Can you
PTAL?

[iannucci, 2016-04-29 20:09:42]

lgtm

https://codereview.chromium.org/1910633006/diff/40001/appengine/cmd/logdog_co...
File appengine/cmd/logdog_coordinator/backend/main.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/cmd/logdog_co...
appengine/cmd/logdog_coordinator/backend/main.go:26: c =
coordinator.UseProdServices(c)
does this re-install stuff like luci/gae? That's actually already in c at the
time you get it here.

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/archival.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/archival.go:61: Project: string(Project(c)),
it's unfortunate that "Project" (as in: working on a project) is the same
spelling as "Project" (as in: projecting video on a wall, or projection queries)

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/config/config.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/config/config.go:176: log.WithError(err).Errorf(c,
"Failed to list 'luci-config' projects.")
TODO: we'll probably want to filter by 'disabled' projects?

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/context.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/context.go:38: // convenience.
the original Context will not be modified or invalidated

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/endpoints/services/registerStream.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/endpoints/services/registerStream.go:221: aeParent,
aeName := archiveExpiredMutation.TaskName(di)
why not just .SetArchiveTaskDeadline(c, m)? In dm I have something like
`mutation.SetExecutionTimeout(c, execution)` which does the whole thing, and
it's a bit easier to work with, since the callsite is always the same oneliner.

[dnj, 2016-04-29 23:04:21]

Thanks for reviewing, updated!

https://codereview.chromium.org/1910633006/diff/40001/appengine/cmd/logdog_co...
File appengine/cmd/logdog_coordinator/backend/main.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/cmd/logdog_co...
appengine/cmd/logdog_coordinator/backend/main.go:26: c =
coordinator.UseProdServices(c)
On 2016/04/29 20:09:42, iannucci wrote:
> does this re-install stuff like luci/gae? That's actually already in c at the
> time you get it here.

Nope, Coordinator services are additional services specific to Coordinator.

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/archival.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/archival.go:61: Project: string(Project(c)),
On 2016/04/29 20:09:42, iannucci wrote:
> it's unfortunate that "Project" (as in: working on a project) is the same
> spelling as "Project" (as in: projecting video on a wall, or projection
queries)

Hah yeah, acknowledged.

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/config/config.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/config/config.go:176: log.WithError(err).Errorf(c,
"Failed to list 'luci-config' projects.")
On 2016/04/29 20:09:42, iannucci wrote:
> TODO: we'll probably want to filter by 'disabled' projects?

Done.

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/context.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/context.go:38: // convenience.
On 2016/04/29 20:09:42, iannucci wrote:
> the original Context will not be modified or invalidated

Done.

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/endpoints/services/registerStream.go (right):

https://codereview.chromium.org/1910633006/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/endpoints/services/registerStream.go:221: aeParent,
aeName := archiveExpiredMutation.TaskName(di)
On 2016/04/29 20:09:42, iannucci wrote:
> why not just .SetArchiveTaskDeadline(c, m)? In dm I have something like
> `mutation.SetExecutionTimeout(c, execution)` which does the whole thing, and
> it's a bit easier to work with, since the callsite is always the same
oneliner.

Discussed, this interface in general needs a face lift, so I'll leave as-is and
carry this forward with the rest of the API.

[dnj, 2016-04-30 01:57:34]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-04-30 01:57:34]

The patchset sent to the CQ was uploaded after l-g-t-m from hinoka@google.com,
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1910633006/#ps60001
(title: "Updated comments.")

[commit-bot: I haz the power, 2016-04-30 01:57:39]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1910633006/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1910633006/60001

[commit-bot: I haz the power, 2016-04-30 02:00:08]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-30 02:00:09]

Try jobs failed on following builders:
  Luci-go Win Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Win%20Tester/...)

[dnj, 2016-04-30 02:07:00]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-04-30 02:07:06]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1910633006/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1910633006/60001

[commit-bot: I haz the power, 2016-04-30 02:10:55]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-30 02:10:57]

Try jobs failed on following builders:
  Luci-go Win Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Win%20Tester/...)

[dnj, 2016-04-30 02:34:04]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-04-30 02:34:05]

The patchset sent to the CQ was uploaded after l-g-t-m from hinoka@google.com,
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1910633006/#ps80001
(title: "Fix authdb tests after gae bump.")

[commit-bot: I haz the power, 2016-04-30 02:34:13]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1910633006/80001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1910633006/80001

[commit-bot: I haz the power, 2016-04-30 02:37:43]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-30 02:37:44]

Try jobs failed on following builders:
  Luci-go Win Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Win%20Tester/...)

[dnj, 2016-04-30 02:41:03]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-04-30 02:41:03]

The patchset sent to the CQ was uploaded after l-g-t-m from hinoka@google.com,
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1910633006/#ps100001
(title: "Update another test.")

[commit-bot: I haz the power, 2016-04-30 02:41:13]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1910633006/100001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1910633006/100001

[commit-bot: I haz the power, 2016-04-30 02:48:53]

Description was changed from

==========
LogDog: Support per-namespace expired archival.

Currently, expired archival only operates on the root namesapce.

Change archival to use named delayed Tumble tasks, scheduling the
cleanup archival on stream registration and clearing it on stream
termination.

BUG=
==========

to

==========
LogDog: Support per-namespace expired archival.

Currently, expired archival only operates on the root namesapce.

Change archival to use named delayed Tumble tasks, scheduling the
cleanup archival on stream registration and clearing it on stream
termination.

BUG=

Committed:
https://github.com/luci/luci-go/commit/4be9524ee752a3d4740a26c0debc1807a2e68f7e
==========

[commit-bot: I haz the power, 2016-04-30 02:48:54]

Committed patchset #6 (id:100001) as
https://github.com/luci/luci-go/commit/4be9524ee752a3d4740a26c0debc1807a2e68f7e