Fix sparse index handling, add index params.

Fix sparse index handling, add index params.

Fix a bug in index archival where sparse indexes without a last log
entry will act as if they have no logs.

Augment the index protobuf with stream-wide parameters, namely the last
prefix and stream indexes in the stream and the total number of entries
in the stream. This will help supplement index information for sparse
indexes. This is also backwards-compatible with indexes without these
parameters.

Finally, add a tool to help query and interact with archived data
records. This was useful in diagnosing this bug, fixing this bug, and
is expected to be useful for future purposes.

BUG=chromium:654880
TEST=local,production
  - Tested against production data. Tool without patch exhibits
    production bug. Tool with patch successfully loads the stream.

Committed: https://github.com/luci/luci-go/commit/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7

[dnj, 2016-10-17 22:12:12]

https://codereview.chromium.org/2422393002/diff/1/logdog/api/logpb/log.proto
File logdog/api/logpb/log.proto (right):

https://codereview.chromium.org/2422393002/diff/1/logdog/api/logpb/log.proto#...
logdog/api/logpb/log.proto:267: /**
New fields, backwards-compatible with old protobufs. All are good to have, only
"last_stream_index" is used ATM.

https://codereview.chromium.org/2422393002/diff/1/logdog/common/archive/archi...
File logdog/common/archive/archive_test.go (right):

https://codereview.chromium.org/2422393002/diff/1/logdog/common/archive/archi...
logdog/common/archive/archive_test.go:86: // indexParams umarshals an index from
a byte stream and removes any entries.
(We now test that the generated index has the right parameters)

https://codereview.chromium.org/2422393002/diff/1/logdog/common/archive/index.go
File logdog/common/archive/index.go (right):

https://codereview.chromium.org/2422393002/diff/1/logdog/common/archive/index...
logdog/common/archive/index.go:32: // Update our stream properties.
(Generate the new protobuf parameters)

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
File logdog/common/storage/archive/logdog_archive_test/main.go (right):

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
logdog/common/storage/archive/logdog_archive_test/main.go:1: // Copyright 2016
The LUCI Authors. All rights reserved.
DON'T review this tool too closely. It's a useful CLI for testing things and
observing things.

I mean, feel free to review if you want *shrug*

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
File logdog/common/storage/archive/storage.go (left):

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
logdog/common/storage/archive/storage.go:322: st.lastIndex =
types.MessageIndex(idx.Entries[len(idx.Entries)-1].StreamIndex)
The actual bug: if there is a sparse index whose last entry, X, ends before the
end of the stream, for a stream with N (>X) entries, and an offset, >X, is
requested, this will falsely assume that the index is contiguous and will say,
"hey, X is the last entry, so there are no logs past X", even though there's
X+1..N.

Now, we only do this optimization if the index contains the cool new "last
stream index" entry.

[dnj, 2016-10-17 22:15:44]

dnj@chromium.org changed reviewers:
+ nodir@chromium.org

[dnj, 2016-10-17 22:15:45]

PTAL

[nodir, 2016-10-17 22:24:07]

lgtm

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
File logdog/common/storage/archive/logdog_archive_test/main.go (right):

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
logdog/common/storage/archive/logdog_archive_test/main.go:1: // Copyright 2016
The LUCI Authors. All rights reserved.
On 2016/10/17 22:12:12, dnj wrote:
> DON'T review this tool too closely. It's a useful CLI for testing things and
> observing things.
> 
> I mean, feel free to review if you want *shrug*

why is it in common? it seems it should be somewhere under logdog/tools or
something

[dnj, 2016-10-17 22:26:19]

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
File logdog/common/storage/archive/logdog_archive_test/main.go (right):

https://codereview.chromium.org/2422393002/diff/1/logdog/common/storage/archi...
logdog/common/storage/archive/logdog_archive_test/main.go:1: // Copyright 2016
The LUCI Authors. All rights reserved.
On 2016/10/17 22:24:06, nodir wrote:
> On 2016/10/17 22:12:12, dnj wrote:
> > DON'T review this tool too closely. It's a useful CLI for testing things and
> > observing things.
> > 
> > I mean, feel free to review if you want *shrug*
> 
> why is it in common? it seems it should be somewhere under logdog/tools or
> something

It's in logdog/common.

[dnj, 2016-10-17 22:26:33]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-10-17 22:26:35]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-18 00:29:39]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-18 00:29:40]

Try jobs failed on following builders:
  Luci-go Win Tester on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/31edcabec673ca10)

[dnj, 2016-10-18 02:03:49]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-10-18 02:03:53]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-18 02:12:09]

Description was changed from

==========
Fix sparse index handling, add index params.

Fix a bug in index archival where sparse indexes without a last log
entry will act as if they have no logs.

Augment the index protobuf with stream-wide parameters, namely the last
prefix and stream indexes in the stream and the total number of entries
in the stream. This will help supplement index information for sparse
indexes. This is also backwards-compatible with indexes without these
parameters.

Finally, add a tool to help query and interact with archived data
records. This was useful in diagnosing this bug, fixing this bug, and
is expected to be useful for future purposes.

BUG=chromium:654880
TEST=local,production
  - Tested against production data. Tool without patch exhibits
    production bug. Tool with patch successfully loads the stream.
==========

to

==========
Fix sparse index handling, add index params.

Fix a bug in index archival where sparse indexes without a last log
entry will act as if they have no logs.

Augment the index protobuf with stream-wide parameters, namely the last
prefix and stream indexes in the stream and the total number of entries
in the stream. This will help supplement index information for sparse
indexes. This is also backwards-compatible with indexes without these
parameters.

Finally, add a tool to help query and interact with archived data
records. This was useful in diagnosing this bug, fixing this bug, and
is expected to be useful for future purposes.

BUG=chromium:654880
TEST=local,production
  - Tested against production data. Tool without patch exhibits
    production bug. Tool with patch successfully loads the stream.

Committed:
https://github.com/luci/luci-go/commit/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7
==========

[commit-bot: I haz the power, 2016-10-18 02:12:10]

Committed patchset #1 (id:1) as
https://github.com/luci/luci-go/commit/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7