LogDog: Update protobufs, add support library.

LogDog: Update protobufs, add support library.

- Add Protobuf support library.
- Add ButlerMetadata message to act as a header frame for published log
  data.
- Published logs from the same Butler instance will now share a secret.
- Explicitly track the index of a stream's terminal log entry.
- Avoid 64-bit integers in protocol, as Javascript hates them.
  - Indices and offsets are now 32-bit.
  - Time is now a RFC3339 string instead of a seconds-from-epoch offset.
  - Time offsets are now a second/nanosecond pair.
- Wrap data blob arrays in a separate message to avoid [][]byte types.
- Add JSON marshal/unmarshal to StreamName type.

R=iannucci@chromium.org

Committed: https://github.com/luci/luci-go/commit/6e402c6cb3f7a3e1032e1941bf2340cb310cb777

[dnj, 2015-08-07 17:06:29]

dnj@chromium.org changed reviewers:
+ iannucci@chromium.org

[dnj, 2015-08-07 17:06:29]

PTAL.

This is a small update to the LogDog protobuf:
- Add "terminal index" field, which lets a message specify the index of the
terminal message. Previously, the terminal index was inferred as the highest log
index in a terminal message. This allows the Butler to send a message without
any log data in the event that a stream closes but the last buffered data of the
stream has already been sent.
- Add "protoutil", a library to operate on protobuf structures.

[tandrii(chromium), 2015-08-11 17:41:36]

tandrii@chromium.org changed reviewers:
+ tandrii@chromium.org

[tandrii(chromium), 2015-08-11 17:41:37]

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/butl...
File common/logdog/protocol/butler.proto (right):

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/butl...
common/logdog/protocol/butler.proto:12: * to describe the remainder of the
contents.
I'm not sure that's the best way to achieve what you want. Maybe ask
protobuf-gurus on our team? I'll let someone else approve this change.

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/prot...
File common/logdog/protocol/protoutil/utils.go (right):

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/prot...
common/logdog/protocol/protoutil/utils.go:4: 
// package protoutil is a library to operate on protobuf structures.

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/prot...
common/logdog/protocol/protoutil/utils.go:33: offset := d.Nanoseconds()
bummer, i didn't expect time.Duration to miss .Microseconds() method.

[dnj, 2015-08-11 18:20:37]

New update includes a few more utility methods.

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/butl...
File common/logdog/protocol/butler.proto (right):

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/butl...
common/logdog/protocol/butler.proto:12: * to describe the remainder of the
contents.
On 2015/08/11 17:41:37, tandrii(chromium) wrote:
> I'm not sure that's the best way to achieve what you want. Maybe ask
> protobuf-gurus on our team? I'll let someone else approve this change.

What do you mean?

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/prot...
File common/logdog/protocol/protoutil/utils.go (right):

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/prot...
common/logdog/protocol/protoutil/utils.go:4: 
On 2015/08/11 17:41:37, tandrii(chromium) wrote:
> // package protoutil is a library to operate on protobuf structures.

Done.

https://codereview.chromium.org/1272893004/diff/1/common/logdog/protocol/prot...
common/logdog/protocol/protoutil/utils.go:33: offset := d.Nanoseconds()
On 2015/08/11 17:41:37, tandrii(chromium) wrote:
> bummer, i didn't expect time.Duration to miss .Microseconds() method.

Acknowledged.

[dnj, 2015-08-26 17:09:13]

dnj@chromium.org changed reviewers:
+ estaab@chromium.org

[dnj, 2015-08-26 17:09:13]

(Ping)

[iannucci, 2015-08-26 18:33:33]

lgtm

https://chromiumcodereview.appspot.com/1272893004/diff/40001/common/logdog/pr...
File common/logdog/protocol/butler.proto (right):

https://chromiumcodereview.appspot.com/1272893004/diff/40001/common/logdog/pr...
common/logdog/protocol/butler.proto:59: * The log stream's secret value
(required).
log prefix's

[dnj, 2015-08-26 18:40:45]

Committed patchset #4 (id:60001) manually as
6e402c6cb3f7a3e1032e1941bf2340cb310cb777 (presubmit successful).