add upload_dir_contents() and download_dir_contents() to gs_utils.py

add upload_dir_contents() and download_dir_contents() to gs_utils.py
(with the ability to set fine-grained ACLs as we need when uploading SKPs)

BUG=skia:2618, skia:1942
R=rmistry@google.com

Committed: https://skia.googlesource.com/common/+/006b9d4

[epoger, 2014-07-18 15:11:16]

Patchset 1 achieves parity with
https://skia.googlesource.com/buildbot/+/master/slave/skia_slave_scripts/util...
(for these particular methods)

Before sending this out for review, I will add the fine-grained ACLs that we
need for uploading SKPs.

[epoger, 2014-07-18 15:48:19]

Ravi- please review at patchset 2.

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py
File py/utils/gs_utils.py (right):

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py#new...
py/utils/gs_utils.py:140: except BotoServerError, e:
While I was in here, improved various error messages.

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py#new...
py/utils/gs_utils.py:146: canned_acl=CANNED_ACL_PRIVATE,
fine_grained_acl_list=None):
I didn't add the http_header_lines param we had in
https://skia.googlesource.com/buildbot/+/master/slave/skia_slave_scripts/util...
, because AFAICT we don't need it.  The HTTP headers are set exactly as we need
them by default (private files are not cached; public files are cached for one
hour).

If we ever start setting them to some other value, we can add the param here.

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py#new...
py/utils/gs_utils.py:206: TODO(epoger): Upload multiple files simultaneously to
reduce latency.
All these TODOs copied in from
https://skia.googlesource.com/buildbot/+/master/slave/skia_slave_scripts/util...

[rmistry, 2014-07-18 16:55:07]

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py
File py/utils/gs_utils.py (right):

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py#new...
py/utils/gs_utils.py:57: CANNED_ACL_PUBLIC_READ_WRITE  = 'public-read-write'
Nit:
Change the above to only have one space before and after the = or align all of
them.

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py#new...
py/utils/gs_utils.py:160: in my manual testing
This should be allowed to be None because if all you care about is the fine
grained ACLs then this is not required. Maybe create  CANNED_ACL_NONE similar to
PERMISSION_NONE. Similarly in upload_dir_contents.

[epoger, 2014-07-18 17:36:19]

Thanks, Ravi...

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py
File py/utils/gs_utils.py (right):

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py#new...
py/utils/gs_utils.py:57: CANNED_ACL_PUBLIC_READ_WRITE  = 'public-read-write'
On 2014/07/18 16:55:07, rmistry wrote:
> Nit:
> Change the above to only have one space before and after the = or align all of
> them.

Aligned 'em all.

https://codereview.chromium.org/407533002/diff/20001/py/utils/gs_utils.py#new...
py/utils/gs_utils.py:160: in my manual testing
On 2014/07/18 16:55:07, rmistry wrote:
> This should be allowed to be None because if all you care about is the fine
> grained ACLs then this is not required. Maybe create  CANNED_ACL_NONE similar
to
> PERMISSION_NONE. Similarly in upload_dir_contents.

I agree, sort of.  Searching for "predefined" in
https://developers.google.com/storage/docs/accesscontrol , it seems to me that
the fine-grained ACLs are always applied on top of *some* "predefined"
("canned") ACL.  Some excerpts:

"If you create a bucket with the default bucket ACLs—that is, you do not specify
a predefined ACL when you create the bucket—your bucket has the predefined
project-private ACL applied to it."

"When you upload an object, you can provide a predefined ACL or not specify an
ACL at all. If you don't specify an ACL, Google Cloud Storage applies the
bucket's default object ACL to the object."

So I have changed the default to None, in which case this object inherits the
bucket's default object ACL.  Whaddaya think?

(Also, I renamed canned_acl as predefined_acl to be consistent with the document
linked to above.)

[rmistry, 2014-07-18 17:38:41]

LGTM

[epoger, 2014-07-18 17:41:03]

Committed patchset #3 manually as r006b9d4 (presubmit successful).