LogDog: Enable keys-only BigTable queries.

LogDog: Enable keys-only BigTable queries.

Currently, since individual log entry records are compacted together
into a RecordIO blob in BigTable, keys-only queries cannot be executed,
since the number of entries in the blob is required in order to traverse
the key space.

This modifies the BigTable schema to encode the number of records in a
row into that row's key. This enables a keys-only query to know this
value and, consequently, traverse the space without needing to load the
full data blob.

A "legacy mode" is supported with the intention of deprecating in a week
or two once uncounted BigTable data has aged off.

BUG=

Committed: https://github.com/luci/luci-go/commit/d0f6fcf838b96874bef8f9c97f9fa095bcff4a1f

[dnj, 2016-04-08 19:52:45]

dnj@chromium.org changed reviewers:
+ estaab@chromium.org, iannucci@chromium.org, vadimsh@chromium.org

[dnj, 2016-04-08 19:52:46]

PTAL

[dnj, 2016-04-11 04:38:00]

Patchset #2 (id:20001) has been deleted

[dnj, 2016-04-13 17:18:43]

dnj@chromium.org changed reviewers:
+ hinoka@chromium.org

[dnj, 2016-04-13 17:18:43]

+hinoka@, PTAL

[Ryan Tseng, 2016-04-13 21:38:05]

hinoka@google.com changed reviewers:
+ hinoka@google.com

[Ryan Tseng, 2016-04-13 21:38:06]

lgtm

how much data would we lose if we didn't bother with a legacy mode?  Just the
in-progress logs right now?  What about just skipping that and deploying at a
low period?

https://codereview.chromium.org/1872903002/diff/60001/server/logdog/storage/b...
File server/logdog/storage/bigtable/rowKey.go (right):

https://codereview.chromium.org/1872903002/diff/60001/server/logdog/storage/b...
server/logdog/storage/bigtable/rowKey.go:30: maxEncodedKeySize =
encodedPrefixSize + (2 * (len("~") + hex.EncodedLen(cmpbin.MaxIntLen64)))
The extra parens seem redundent

[dnj, 2016-04-13 21:44:05]

> how much data would we lose if we didn't bother with a legacy mode?  Just the
> in-progress logs right now?  What about just skipping that and deploying at a
> low period?

We lose any unarchived logs, which would be:
(a) all logs that have been collected before but not archived yet, and
(b) all logs that can't be archived b/c they are not terminated.

We could skip them, but this CL is deployed on prod ATM and everything's fine,
so I'm going to just go with it.

> 
>
https://codereview.chromium.org/1872903002/diff/60001/server/logdog/storage/b...
> File server/logdog/storage/bigtable/rowKey.go (right):
> 
>
https://codereview.chromium.org/1872903002/diff/60001/server/logdog/storage/b...
> server/logdog/storage/bigtable/rowKey.go:30: maxEncodedKeySize =
> encodedPrefixSize + (2 * (len("~") + hex.EncodedLen(cmpbin.MaxIntLen64)))
> The extra parens seem redundent

[dnj, 2016-04-13 21:44:15]

https://codereview.chromium.org/1872903002/diff/60001/server/logdog/storage/b...
File server/logdog/storage/bigtable/rowKey.go (right):

https://codereview.chromium.org/1872903002/diff/60001/server/logdog/storage/b...
server/logdog/storage/bigtable/rowKey.go:30: maxEncodedKeySize =
encodedPrefixSize + (2 * (len("~") + hex.EncodedLen(cmpbin.MaxIntLen64)))
On 2016/04/13 21:38:06, Ryan Tseng wrote:
> The extra parens seem redundent

Acknowledged.

[dnj, 2016-04-20 18:41:28]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-04-20 18:41:36]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1872903002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1872903002/60001

[commit-bot: I haz the power, 2016-04-20 18:45:30]

Description was changed from

==========
LogDog: Enable keys-only BigTable queries.

Currently, since individual log entry records are compacted together
into a RecordIO blob in BigTable, keys-only queries cannot be executed,
since the number of entries in the blob is required in order to traverse
the key space.

This modifies the BigTable schema to encode the number of records in a
row into that row's key. This enables a keys-only query to know this
value and, consequently, traverse the space without needing to load the
full data blob.

A "legacy mode" is supported with the intention of deprecating in a week
or two once uncounted BigTable data has aged off.

BUG=
==========

to

==========
LogDog: Enable keys-only BigTable queries.

Currently, since individual log entry records are compacted together
into a RecordIO blob in BigTable, keys-only queries cannot be executed,
since the number of entries in the blob is required in order to traverse
the key space.

This modifies the BigTable schema to encode the number of records in a
row into that row's key. This enables a keys-only query to know this
value and, consequently, traverse the space without needing to load the
full data blob.

A "legacy mode" is supported with the intention of deprecating in a week
or two once uncounted BigTable data has aged off.

BUG=

Committed:
https://github.com/luci/luci-go/commit/d0f6fcf838b96874bef8f9c97f9fa095bcff4a1f
==========

[commit-bot: I haz the power, 2016-04-20 18:45:31]

Committed patchset #3 (id:60001) as
https://github.com/luci/luci-go/commit/d0f6fcf838b96874bef8f9c97f9fa095bcff4a1f