Archivist asserts completeness through keys scan.

Archivist asserts completeness through keys scan.

Currently, when the Archivist needs to archive a complete stream, it
attempts to do an archival and fails if records are missing. This is
fairly expensive, as the full archival requires Google Storage
operations and potentially large amounts of Intermediate Storage data
being pulled. A bad case is when the missing log entry is the last one
in a very long stream.

This takes advantage of the new Storage keys-only Get support to assert
completeness up-front via a keys-only scan of the log stream. This
improves archival speeds in failure cases and simplifies archival logic.

BUG=

Committed: https://github.com/luci/luci-go/commit/98a579258ed1e62f2a70edd718e947c08eb6ddee

[dnj, 2016-04-08 19:52:55]

dnj@chromium.org changed reviewers:
+ estaab@chromium.org, iannucci@chromium.org, vadimsh@chromium.org

[dnj, 2016-04-08 19:52:55]

PTAL

[iannucci, 2016-04-14 01:13:41]

lgtm

[dnj, 2016-04-20 18:46:06]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-04-20 18:46:06]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1874563005/#ps40001
(title: "Remove unused members.")

[commit-bot: I haz the power, 2016-04-20 18:46:16]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1874563005/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1874563005/40001

[commit-bot: I haz the power, 2016-04-20 18:52:49]

Description was changed from

==========
Archivist asserts completeness through keys scan.

Currently, when the Archivist needs to archive a complete stream, it
attempts to do an archival and fails if records are missing. This is
fairly expensive, as the full archival requires Google Storage
operations and potentially large amounts of Intermediate Storage data
being pulled. A bad case is when the missing log entry is the last one
in a very long stream.

This takes advantage of the new Storage keys-only Get support to assert
completeness up-front via a keys-only scan of the log stream. This
improves archival speeds in failure cases and simplifies archival logic.

BUG=
==========

to

==========
Archivist asserts completeness through keys scan.

Currently, when the Archivist needs to archive a complete stream, it
attempts to do an archival and fails if records are missing. This is
fairly expensive, as the full archival requires Google Storage
operations and potentially large amounts of Intermediate Storage data
being pulled. A bad case is when the missing log entry is the last one
in a very long stream.

This takes advantage of the new Storage keys-only Get support to assert
completeness up-front via a keys-only scan of the log stream. This
improves archival speeds in failure cases and simplifies archival logic.

BUG=

Committed:
https://github.com/luci/luci-go/commit/98a579258ed1e62f2a70edd718e947c08eb6ddee
==========

[commit-bot: I haz the power, 2016-04-20 18:52:50]

Committed patchset #3 (id:40001) as
https://github.com/luci/luci-go/commit/98a579258ed1e62f2a70edd718e947c08eb6ddee