WIP: Archive command which is much faster

WIP: Archive command which is much faster

 * Uploads small files inside an ar archive.
 * Uses a better file system walking method.
 * Keeps all reads linear.
 * Uses a more efficient pipelining system.
 * Doesn't do expensive stat calls.

BUG=598990

[tandrii_google, 2016-06-06 08:37:51]

tandrii@google.com changed reviewers:
+ tandrii@google.com

[tandrii_google, 2016-06-06 08:37:52]

is it correct, that with this CL, if I am archiving two smallish files inside
the same directory, say d/a and d/b, the resulting isolate will actually contain
smth like d/smallishfilesarchive-1 which the isolate client on the bot will have
to understand and un-archive appropriately?

[mithro, 2016-06-06 16:33:29]

On 2016/06/06 at 08:37:52, tandrii wrote:
> is it correct, that with this CL, if I am archiving two smallish files inside
the same directory, say d/a and d/b, the resulting isolate will actually contain
smth like d/smallishfilesarchive-1 which the isolate client on the bot will have
to understand and un-archive appropriately?

Yes. The unarchive side is almost complete too (for the moment I just have proof
of concept Python prescript I run on the isolate). I have tested and extraction
is also very fast.

[mithro, 2016-06-06 16:38:33]

On 2016/06/06 at 16:33:29, mithro wrote:
> On 2016/06/06 at 08:37:52, tandrii wrote:
> > is it correct, that with this CL, if I am archiving two smallish files
inside the same directory, say d/a and d/b, the resulting isolate will actually
contain smth like d/smallishfilesarchive-1 which the isolate client on the bot
will have to understand and un-archive appropriately?
> 
> Yes. The unarchive side is almost complete too (for the moment I just have
proof of concept Python prescript I run on the isolate). I have tested and
extraction is also very fast.

There is some discussion about the performance of this code at
https://github.com/luci/luci-go/issues/9

As discussed with Maruel, this patch is being split into three parts;

 (a) The archive manipulation code (which is at
https://codereview.chromium.org/2043623002/).
 (b) The fast walk / dirtools code
 (c) The network pipeline rework + fastarchive command.

There will also be a fourth patch which does;

 (d) The download side

The download side will need to be done in both luci-go and luci-py, but luckily
all my proof of concept code was in Python, so that is pretty easy to do.

Tim 'mithro' Ansell

[M-A Ruel, 2016-06-08 20:52:35]

maruel@chromium.org changed reviewers:
+ maruel@chromium.org

[M-A Ruel, 2016-06-08 20:52:36]

skipping ar and dirtools

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
File client/cmd/isolate/fastarchive.go (right):

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
client/cmd/isolate/fastarchive.go:18: //"github.com/luci/luci-go/client/isolate"
please trim comments

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
client/cmd/isolate/fastarchive.go:34: type ToHash struct {
these do not need to be exported

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
client/cmd/isolate/fastarchive.go:62: const CHECK_BATCH_SIZE = 20
I prefer const to be at the top for consistency, it's not a hard requirement

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
client/cmd/isolate/fastarchive.go:64: func ChckFile(is
isolatedclient.IsolateServer, canceler common.Canceler, src <-chan *ToCheck, dst
chan<- *ToPush) {
Check?

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
client/cmd/isolate/fastarchive.go:119: close(dst)
don't close a channel handed in, have the caller close it.

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
client/cmd/isolate/fastarchive.go:224: done_chan := make(chan bool)
technically, you want a sync.WaitGroup so in each function, you'd do defer
wg.Done() and do a wg.Wait() at the end

https://codereview.chromium.org/2014243002/diff/60001/client/cmd/isolate/fast...
client/cmd/isolate/fastarchive.go:228: go HashFile(is, canceler, hash_chan,
chck_chan)
- these do not need to be exported
- I prefer the close() call to be here, not inside the function. This makes the
code here more clunky but on the other hand ownership is clear, e.g.

go func() {
  defer close(chk_chan)
  defer wg.Done()
  hashFile(is, canceler, hash_chan, chk_chan)
}()


- mind using a stageNfooBar form, with N being the stage number? It makes
scanning the functions easier to relate to the logical ordering.