LogDog: BigTable batching schema.

LogDog: BigTable batching schema.

Update the Storage schema to enable batching Put() calls. Update
BigTable schema to store batches of sequential logs in a single cell
instead of requiring one cell per log.

Previously, BigTable schema keyed each cell row off of a log's index.
Now, the cell row is keyed based on the LAST index in a block of
sequential log entries, storing entries as a series of recordio blocks.
This change allows multiple log entries to be stored in a single
BigTable cell.

  Before: 0|1|2|3|4|5
  Now:    0 1 2 3|4 5

Before, "get message 2" meant "go to row #2, get data." Now, it is "go
to row >= 2 (in this example, 3) and pull 2 out of the block.

Update APIs accordingly.

Also update Subscriber API to control message quantity rather than
batch pull count for better tuning and parity with upstream Pub/Sub
API changes.

BUG=chromium:597886
TEST=local

Committed: https://github.com/luci/luci-go/commit/63c19cda88aa254fd537edc802bf5c19cc8f009f

[dnj, 2016-03-28 15:24:33]

dnj@chromium.org changed reviewers:
+ estaab@chromium.org, iannucci@chromium.org

[dnj, 2016-03-28 15:24:34]

PTAL

[dnj, 2016-03-29 17:27:09]

(Ping!)

[dnj, 2016-03-29 19:29:45]

dnj@chromium.org changed reviewers:
+ nodir@chromium.org, vadimsh@chromium.org

[dnj, 2016-03-29 19:29:46]

+some more people, since estaab@ is out of country and iannucci@ is fairly busy
ATM.

[nodir, 2016-03-30 18:29:42]

lgtm

what do you with rows that have old schema?

https://codereview.chromium.org/1838803002/diff/1/common/gcloud/pubsub/subscr...
File common/gcloud/pubsub/subscriber/subscriber.go (right):

https://codereview.chromium.org/1838803002/diff/1/common/gcloud/pubsub/subscr...
common/gcloud/pubsub/subscriber/subscriber.go:1: // Copyright 2016 The Chromium
Authors. All rights reserved.
I assume this code will be replaced by the new pubsub lib

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
File server/logdog/storage/bigtable/bigtable.go (left):

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
server/logdog/storage/bigtable/bigtable.go:34: type btGetCallback func(*rowKey,
[]byte) error
[general feedback] you don't have to prefix types with bt in bigtable package

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
File server/logdog/storage/bigtable/storage.go (right):

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
server/logdog/storage/bigtable/storage.go:85: // for Storage calls). See GAE
code for more information.
"GAE code" is abstract. Provide a link or say something like bigtable clients
store it, so I have to too

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
server/logdog/storage/bigtable/storage.go:93: maxRowSize int
a comment would be nice. Initially I thought it is about "row count", not "byte
count"

[dnj, 2016-03-30 18:37:14]

Previous BT data has been trashed, which I'm very okay with.

https://codereview.chromium.org/1838803002/diff/1/common/gcloud/pubsub/subscr...
File common/gcloud/pubsub/subscriber/subscriber.go (right):

https://codereview.chromium.org/1838803002/diff/1/common/gcloud/pubsub/subscr...
common/gcloud/pubsub/subscriber/subscriber.go:1: // Copyright 2016 The Chromium
Authors. All rights reserved.
On 2016/03/30 18:29:42, nodir wrote:
> I assume this code will be replaced by the new pubsub lib

Yep, this is the follow-on CL that I will submit immediately after this:
https://codereview.chromium.org/1838303002/

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
File server/logdog/storage/bigtable/bigtable.go (left):

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
server/logdog/storage/bigtable/bigtable.go:34: type btGetCallback func(*rowKey,
[]byte) error
On 2016/03/30 18:29:42, nodir wrote:
> [general feedback] you don't have to prefix types with bt in bigtable package

Yep, but in this case, this is a BigTable-specific binding, as opposed to the
rest of the code, which is a higher level abstraction on top of the binding for
testing.

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
File server/logdog/storage/bigtable/storage.go (right):

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
server/logdog/storage/bigtable/storage.go:85: // for Storage calls). See GAE
code for more information.
On 2016/03/30 18:29:42, nodir wrote:
> "GAE code" is abstract. Provide a link or say something like bigtable clients
> store it, so I have to too

Done.

https://codereview.chromium.org/1838803002/diff/1/server/logdog/storage/bigta...
server/logdog/storage/bigtable/storage.go:93: maxRowSize int
On 2016/03/30 18:29:42, nodir wrote:
> a comment would be nice. Initially I thought it is about "row count", not
"byte
> count"

Done.

[dnj, 2016-04-01 22:16:11]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-04-01 22:16:11]

The patchset sent to the CQ was uploaded after l-g-t-m from nodir@chromium.org
Link to the patchset: https://codereview.chromium.org/1838803002/#ps20001
(title: "Minor comments and quality of code tweaks.")

[commit-bot: I haz the power, 2016-04-01 22:16:15]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1838803002/20001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1838803002/20001

[commit-bot: I haz the power, 2016-04-01 22:19:57]

Description was changed from

==========
LogDog: BigTable batching schema.

Update the Storage schema to enable batching Put() calls. Update
BigTable schema to store batches of sequential logs in a single cell
instead of requiring one cell per log.

Previously, BigTable schema keyed each cell row off of a log's index.
Now, the cell row is keyed based on the LAST index in a block of
sequential log entries, storing entries as a series of recordio blocks.
This change allows multiple log entries to be stored in a single
BigTable cell.

  Before: 0|1|2|3|4|5
  Now:    0 1 2 3|4 5

Before, "get message 2" meant "go to row #2, get data." Now, it is "go
to row >= 2 (in this example, 3) and pull 2 out of the block.

Update APIs accordingly.

Also update Subscriber API to control message quantity rather than
batch pull count for better tuning and parity with upstream Pub/Sub
API changes.

BUG=chromium:597886
TEST=local
==========

to

==========
LogDog: BigTable batching schema.

Update the Storage schema to enable batching Put() calls. Update
BigTable schema to store batches of sequential logs in a single cell
instead of requiring one cell per log.

Previously, BigTable schema keyed each cell row off of a log's index.
Now, the cell row is keyed based on the LAST index in a block of
sequential log entries, storing entries as a series of recordio blocks.
This change allows multiple log entries to be stored in a single
BigTable cell.

  Before: 0|1|2|3|4|5
  Now:    0 1 2 3|4 5

Before, "get message 2" meant "go to row #2, get data." Now, it is "go
to row >= 2 (in this example, 3) and pull 2 out of the block.

Update APIs accordingly.

Also update Subscriber API to control message quantity rather than
batch pull count for better tuning and parity with upstream Pub/Sub
API changes.

BUG=chromium:597886
TEST=local

Committed:
https://github.com/luci/luci-go/commit/63c19cda88aa254fd537edc802bf5c19cc8f009f
==========

[commit-bot: I haz the power, 2016-04-01 22:19:58]

Committed patchset #2 (id:20001) as
https://github.com/luci/luci-go/commit/63c19cda88aa254fd537edc802bf5c19cc8f009f