luci-py/isolateserver.py: Add archive support when downloading.

appengine/isolate/doc/Design.md

 # Isolated Server
 
 High performance Infrastructure to run an executable with all its dependencies
 mapped in.
 
 **One line description:** How to push 10000 files worth 2Gb in total to tens of
 bots within a few seconds, *even on Windows or Android*.
 
 
 ## Objective
@@ skipping 99 matching lines @@
 The root is a dictionary with the following keys:
   - `algo`: Hashing algorithm used to hash the content. Normally `'sha-1'`.
   - `command`: exact command to run as a list.
   - `files`: list of dictionary, each key being the relative file path, and the
     entry being a dict determining the properties of the file. Exactly one of
     `h` or `l` must be present. `m` must be present only on POSIX systems.
     - `h`: file content's SHA-1
     - `l`: link destination iff a symlink
     - `m`: posix file mode iff on posix
     - `s`: file size iff not a symlink
+    - `t`: type of the file, defaults to `basic` if not given.

[M-A Ruel, 2016/06/22 17:17:49]

I'd make the statement stronger, it should not be given if the default.

[mithro, 2016/06/23 07:17:21]

Done.

   - `includes`: references another `.isolated` file for additional files or to
     provide the command. In practice, this is used to reduce `.isolated` file
     size by moving rarely changed test data files in a separate `.isolated`
     file.
   - `read_only`: boolean to specify if all the files should be read-only. This
     will be eventually enforced.
   - `relative_cwd`: relative directory inside the temporary directory tree when
     the command should be executed from.
   - `version`: version of the file format. Increment the minor version for non
     breaking changes and the major version if code written for a previous
     version shouldn't be able to parse it.
 
 
+##### File types
+
+There are two file types;
+  - `basic`: All normal files, the default type.
+  - `smallfiles-archive`: An [ar](https://en.wikipedia.org/wiki/Ar_(Unix))

[M-A Ruel, 2016/06/22 17:17:49]

this is overly verbose. :/  I think 'ar' is sufficient.

[mithro, 2016/06/23 07:17:21]

How about "smallar"? If we want to use ar files in the future for different
reason it would be good to have a way to differentiate.

[M-A Ruel, 2016/06/23 13:29:01]

I don't understand, it's an 'ar' file. We use 'ar' files here. If we use ar
files for another reason later, we'll use another value but for now, the file
type is 'ar'.

[mithro, 2016/07/28 16:38:00]

Done.

+    archive containing a large number of small files.
+
+
 #### Arbitrary split vs recursive trees
 
 The `.isolated` format supports the `includes` key to split and merge back list
 of files in separate `.isolated` files. It is in stark contrast with more
 traditional tree of trees structure like the git tree object rooted to a git
 commit object.
 
 The reason is to leave a lot of room to the tool generating the `.isolated`
 files to be able to package low-churn files versus high-churn files into a small
 number of `.isolated` files. As a practical example, we can state that test data
@@ skipping 221 matching lines @@
 
 ## Testing Plan
 
   - The isolate server code is unit, smoke and canary tested. Since most of the
     isolate server code is OS-independent and written in python, testing is
     relatively easy.
   - Support for hardlinks, symlinks and native path case need OS-specific code
     which can be tested itself on Swarming to get coverage across OSes.
   - A canary Continuous Integration master is run by the chromium team at
     http://build.chromium.org/p/chromium.swarm/waterfall.