Iterate archive query alongside task queue.

Iterate archive query alongside task queue.

Currently, Coordinator performs one big query to get all archive tasks,
then archives them sequentially. That query may fail if it's too big,
though. This change makes it so the Coordinator performs an iterative
query and enqueues tasks iteratively.

BUG=

Committed: https://github.com/luci/luci-go/commit/5467eab0823ac292b84322b7cf8277fb69c96be1

[dnj, 2016-03-30 17:04:21]

dnj@chromium.org changed reviewers:
+ estaab@chromium.org, iannucci@chromium.org

[dnj, 2016-03-30 17:04:22]

PTAL

[dnj, 2016-03-30 18:40:06]

dnj@chromium.org changed reviewers:
+ nodir@chromium.org

[dnj, 2016-03-30 18:40:07]

+nodir@, if you have time.

[iannucci, 2016-03-30 19:04:28]

https://codereview.chromium.org/1844963002/diff/1/appengine/logdog/coordinato...
File appengine/logdog/coordinator/backend/archiveCron.go (right):

https://codereview.chromium.org/1844963002/diff/1/appengine/logdog/coordinato...
appengine/logdog/coordinator/backend/archiveCron.go:152: err = di.Run(iterQuery,
func(ls *coordinator.LogStream, cb ds.CursorCB) error {
I would just loop in the query and dispatch the tasks from inside the query body
as batches.

[dnj, 2016-03-30 19:10:46]

https://codereview.chromium.org/1844963002/diff/1/appengine/logdog/coordinato...
File appengine/logdog/coordinator/backend/archiveCron.go (right):

https://codereview.chromium.org/1844963002/diff/1/appengine/logdog/coordinato...
appengine/logdog/coordinator/backend/archiveCron.go:152: err = di.Run(iterQuery,
func(ls *coordinator.LogStream, cb ds.CursorCB) error {
On 2016/03/30 19:04:27, iannucci wrote:
> I would just loop in the query and dispatch the tasks from inside the query
body
> as batches.

Done.

[dnj, 2016-03-31 01:12:19]

Updated to use new query batching filter, blocked on:
https://chromiumcodereview.appspot.com/1843313004/

[iannucci, 2016-03-31 22:52:31]

lgtm

https://chromiumcodereview.appspot.com/1844963002/diff/40001/appengine/logdog...
File appengine/logdog/coordinator/backend/archiveCron.go (right):

https://chromiumcodereview.appspot.com/1844963002/diff/40001/appengine/logdog...
appengine/logdog/coordinator/backend/archiveCron.go:37: return
fmt.Sprintf("archive--%s", hashID)
let's put a version number in here too

or undo this

https://chromiumcodereview.appspot.com/1844963002/diff/40001/appengine/logdog...
appengine/logdog/coordinator/backend/archiveCron.go:165: tasks = tasks[:0]
would it be cleaner to do this prune at the callsite?

https://chromiumcodereview.appspot.com/1844963002/diff/40001/appengine/logdog...
appengine/logdog/coordinator/backend/archiveCron.go:170: err = ds.Get(c).Run(q,
func(ls *coordinator.LogStream) error {
dsS := ds.Get(dsQueryBatch.......)

or something

[dnj, 2016-04-01 00:21:52]

https://codereview.chromium.org/1844963002/diff/40001/appengine/logdog/coordi...
File appengine/logdog/coordinator/backend/archiveCron.go (right):

https://codereview.chromium.org/1844963002/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/backend/archiveCron.go:37: return
fmt.Sprintf("archive--%s", hashID)
On 2016/03/31 22:52:31, iannucci wrote:
> let's put a version number in here too
> 
> or undo this

Done.

https://codereview.chromium.org/1844963002/diff/40001/appengine/logdog/coordi...
appengine/logdog/coordinator/backend/archiveCron.go:165: tasks = tasks[:0]
On 2016/03/31 22:52:31, iannucci wrote:
> would it be cleaner to do this prune at the callsite?

I don't think so. Since this manages the "tasks" array, it makes sense to me
that it would clear it. I will make this function fully manage the array,
though.

[dnj, 2016-04-01 00:24:29]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-04-01 00:24:30]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1844963002/#ps60001
(title: "Respond to code review comments.")

[commit-bot: I haz the power, 2016-04-01 00:24:31]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-01 00:24:32]

This CL has an open dependency (Issue 1846053002 Patch 1). Please resolve the
dependency and try again.

[dnj, 2016-04-01 00:35:55]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-04-01 00:36:04]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1844963002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1844963002/60001

[commit-bot: I haz the power, 2016-04-01 00:37:33]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-01 00:37:33]

Try jobs failed on following builders:
  Luci-go Linux Trusty 64 Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Linux%20Trust...)

[dnj, 2016-04-01 01:40:11]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-04-01 01:40:18]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1844963002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1844963002/60001

[commit-bot: I haz the power, 2016-04-01 01:43:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-01 01:43:25]

Try jobs failed on following builders:
  Luci-go Win Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Win%20Tester/...)

[dnj, 2016-04-01 01:45:22]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-04-01 01:45:24]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1844963002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1844963002/60001

[commit-bot: I haz the power, 2016-04-01 01:50:19]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-01 01:50:20]

Try jobs failed on following builders:
  Luci-go Win Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Win%20Tester/...)

[dnj, 2016-04-01 22:12:02]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-04-01 22:12:06]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1844963002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1844963002/60001

[commit-bot: I haz the power, 2016-04-01 22:15:46]

Description was changed from

==========
Iterate archive query alongside task queue.

Currently, Coordinator performs one big query to get all archive tasks,
then archives them sequentially. That query may fail if it's too big,
though. This change makes it so the Coordinator performs an iterative
query and enqueues tasks iteratively.

BUG=
==========

to

==========
Iterate archive query alongside task queue.

Currently, Coordinator performs one big query to get all archive tasks,
then archives them sequentially. That query may fail if it's too big,
though. This change makes it so the Coordinator performs an iterative
query and enqueues tasks iteratively.

BUG=

Committed:
https://github.com/luci/luci-go/commit/5467eab0823ac292b84322b7cf8277fb69c96be1
==========

[commit-bot: I haz the power, 2016-04-01 22:15:47]

Committed patchset #4 (id:60001) as
https://github.com/luci/luci-go/commit/5467eab0823ac292b84322b7cf8277fb69c96be1