In upload_to_google_storage, pass -z argument through to gsutil.

In upload_to_google_storage, pass -z argument through to gsutil.

Also fix some latent bugs in the unit tests.

BUG=467005

[azarchs, 2015-03-31 13:36:19]

azarchs@chromium.org changed reviewers:
+ jochen@chromium.org, pasko@chromium.org

[jochen (gone - plz use gerrit), 2015-04-01 14:38:02]

i don't know about the google storage stuff, so I'm not a good reviewer for this

[azarchs, 2015-04-01 14:41:20]

On 2015/04/01 14:38:02, jochen wrote:
> i don't know about the google storage stuff, so I'm not a good reviewer for
this

Ok.  Just picked you because you're the only OWNER in EMEA.  I'll find someone
else.

[azarchs, 2015-04-01 14:42:52]

azarchs@chromium.org changed reviewers:
+ hinoka@chromium.org
- jochen@chromium.org

[pasko, 2015-04-01 18:41:39]

this is handy, non-owner lgtm, thank you, Adam!

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py
File upload_to_google_storage.py (right):

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py...
upload_to_google_storage.py:233: help='Gzip files which end in ext. '
nit: it took me a bit of time to get the intuition of what 'ext' is, I think
'suffix' would be more readable.

nit: s/may be/is/

[azarchs, 2015-04-02 09:47:23]

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py
File upload_to_google_storage.py (right):

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py...
upload_to_google_storage.py:233: help='Gzip files which end in ext. '
On 2015/04/01 18:41:39, pasko wrote:
> nit: it took me a bit of time to get the intuition of what 'ext' is, I think
> 'suffix' would be more readable.


It's standard terminology in windows to call the suffix the file extension. 
More importantly, ext is what it's called in the documentation for gsutil.  For
consistency's sake I think it's better to keep it that way.

> 
> nit: s/may be/is/
Done.

[hinoka, 2015-04-03 09:59:32]

Lgtm

I'd also be okay with whitelisting various extensions to be gzipped by default

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py
File upload_to_google_storage.py (right):

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py...
upload_to_google_storage.py:97: gsutil_args.append('-z')
nit: use .extend()

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py...
upload_to_google_storage.py:232: parser.add_option('-z', dest='gzip',
metavar='ext',
I prefer ('-z', '--gzip', ...), in which case dest is implied.

[azarchs, 2015-04-03 10:07:21]

Thanks.  For now I think leaving the behavior unchanged is fine.  Whitelisting
some extensions would be as simple as setting a default for the parser. 
Unnecessary for my use case, and I don't know off the top of head what the use
cases are and what the whitelist should be.

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py
File upload_to_google_storage.py (right):

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py...
upload_to_google_storage.py:97: gsutil_args.append('-z')
On 2015/04/03 09:59:31, hinoka wrote:
> nit: use .extend()

Done.

https://codereview.chromium.org/1048103002/diff/1/upload_to_google_storage.py...
upload_to_google_storage.py:232: parser.add_option('-z', dest='gzip',
metavar='ext',
On 2015/04/03 09:59:31, hinoka wrote:
> I prefer ('-z', '--gzip', ...), in which case dest is implied.

Done.

[azarchs, 2015-04-03 10:07:44]

The CQ bit was checked by azarchs@chromium.org

[azarchs, 2015-04-03 10:07:44]

The patchset sent to the CQ was uploaded after l-g-t-m from pasko@chromium.org,
hinoka@chromium.org
Link to the patchset: https://codereview.chromium.org/1048103002/#ps40001
(title: "Address comments from hinoka")

[commit-bot: I haz the power, 2015-04-03 10:07:51]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1048103002/40001

[commit-bot: I haz the power, 2015-04-03 10:12:35]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-03 10:12:36]

Presubmit check for 1048103002-40001 failed and returned exit status 1.

Running presubmit commit checks ...
transaction abort!
rollback completed
abort: connection ended unexpectedly
Checking out rietveld...

** Presubmit ERRORS **
Failed to checkout rietveld. Do you have mercurial installed?
Command hg clone -q -u 9349cab9a3bb -r 9349cab9a3bb
https://code.google.com/p/rietveld/
/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/_rietveld
returned non-zero exit status 255

Presubmit checks took 277.9s to calculate.

[azarchs, 2015-04-03 11:43:03]

The CQ bit was checked by azarchs@chromium.org

[commit-bot: I haz the power, 2015-04-03 11:43:08]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1048103002/40001

[commit-bot: I haz the power, 2015-04-03 11:47:33]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-03 11:47:34]

Presubmit check for 1048103002-40001 failed and returned exit status 1.

Running presubmit commit checks ...
transaction abort!
rollback completed
abort: connection ended unexpectedly
Checking out rietveld...

** Presubmit ERRORS **
tests/gclient_smoketest.py (53.80s) failed
Ffatal: reference is not a tree: 9a46fd240fc767610d883760a79dbd4d71f35654
EE..............................................
======================================================================
ERROR: testBlinkDEPSChangeUsingGit (__main__.BlinkDEPSTransitionSmokeTest)
Like testBlinkDEPSChangeUsingGclient, but move the main project using
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1630, in testBlinkDEPSChangeUsingGit
    cwd=self.checkout_path)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
484, in check_call
    check_call_out(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
478, in check_call_out
    returncode, args, kwargs.get('cwd'), out[0], out[1])
CalledProcessError: Command git checkout -q
9a46fd240fc767610d883760a79dbd4d71f35654 returned non-zero exit status 128 in
/tmp/trialNRvyRn/__main__.BlinkDEPSTransitionSmokeTest.testBlinkDEPSChangeUsingGit/src

======================================================================
ERROR: testBlinkLocalBranchesArePreserved
(__main__.BlinkDEPSTransitionSmokeTest)
Checks that the state of local git branches are effectively preserved
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1656, in
testBlinkLocalBranchesArePreserved
    self.CheckStatusPreMergePoint()
  File "tests/gclient_smoketest.py", line 1558, in CheckStatusPreMergePoint
    self.blink), self.blink_git_url)
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/scm.py", line
121, in Capture
    cwd=cwd, stderr=subprocess2.PIPE, env=env, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
515, in check_output
    return check_call_out(args, stdout=PIPE, **kwargs)[0]
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
475, in check_call_out
    out, returncode = communicate(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
449, in communicate
    proc = Popen(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
253, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory:
'/tmp/trialNRvyRn/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit'.
Check that
/tmp/trialNRvyRn/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit
or git exist and have execution permission.

======================================================================
FAIL: testBlinkDEPSChangeUsingGclient (__main__.BlinkDEPSTransitionSmokeTest)
Checks that {src,blink} repos are consistent when syncing going back and
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1601, in
testBlinkDEPSChangeUsingGclient
    self.assertEqual(res[2], 0, 'DEPS change sync failed.')
AssertionError: DEPS change sync failed.

----------------------------------------------------------------------
Ran 49 tests in 53.641s

FAILED (failures=1, errors=2)


Failed to checkout rietveld. Do you have mercurial installed?
Command hg clone -q -u 9349cab9a3bb -r 9349cab9a3bb
https://code.google.com/p/rietveld/
/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/_rietveld
returned non-zero exit status 255

Presubmit checks took 259.1s to calculate.

[azarchs, 2015-04-03 12:54:06]

The CQ bit was checked by azarchs@chromium.org

[commit-bot: I haz the power, 2015-04-03 12:54:18]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1048103002/40001

[commit-bot: I haz the power, 2015-04-03 12:56:40]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-03 12:56:41]

Presubmit check for 1048103002-40001 failed and returned exit status 1.

Running presubmit commit checks ...
** unknown exception encountered, please report by visiting
** http://mercurial.selenic.com/wiki/BugTracker
** Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2]
** Mercurial Distributed SCM (version 2.8.2)
** Extensions loaded: 
Traceback (most recent call last):
  File "/usr/bin/hg", line 38, in <module>
    mercurial.dispatch.run()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 28, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 69, in
dispatch
    ret = _runcatch(req)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 133, in
_runcatch
    return _dispatch(req)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 806, in
_dispatch
    cmdpats, cmdoptions)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 585, in
runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 897, in
_runcommand
    return checkargs()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 868, in
checkargs
    return cmdfunc()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 803, in
<lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 512, in check
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mercurial/commands.py", line 1286, in
clone
    branch=opts.get('branch'))
  File "/usr/lib/python2.7/dist-packages/mercurial/hg.py", line 369, in clone
    revs = [srcpeer.lookup(r) for r in rev]
  File "/usr/lib/python2.7/dist-packages/mercurial/wireproto.py", line 115, in
plain
    encresref.set(self._submitone(f.func_name, encargsorres))
  File "/usr/lib/python2.7/dist-packages/mercurial/wireproto.py", line 163, in
_submitone
    return self._call(op, **args)
  File "/usr/lib/python2.7/dist-packages/mercurial/httppeer.py", line 173, in
_call
    return fp.read()
  File "/usr/lib/python2.7/dist-packages/mercurial/keepalive.py", line 422, in
read
    s = self._rbuf + self._raw_read(amt)
  File "/usr/lib/python2.7/httplib.py", line 543, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/dist-packages/mercurial/keepalive.py", line 445, in
_read_chunked
    raise httplib.IncompleteRead(value)
httplib.IncompleteRead: IncompleteRead(31 bytes read)
Checking out rietveld...

** Presubmit ERRORS **
Failed to checkout rietveld. Do you have mercurial installed?
Command hg clone -q -u 9349cab9a3bb -r 9349cab9a3bb
https://code.google.com/p/rietveld/
/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/_rietveld
returned non-zero exit status 1

Presubmit checks took 135.5s to calculate.

[azarchs, 2015-04-03 13:24:00]

These presubmit failures appear to be unrelated to this change.  I see no option
besides continuing to retry until it works.

[azarchs, 2015-04-03 13:24:05]

The CQ bit was checked by azarchs@chromium.org

[commit-bot: I haz the power, 2015-04-03 13:24:13]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1048103002/40001

[commit-bot: I haz the power, 2015-04-03 13:29:11]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-03 13:29:12]

Presubmit check for 1048103002-40001 failed and returned exit status 1.

Running presubmit commit checks ...
transaction abort!
rollback completed
abort: connection ended unexpectedly
Checking out rietveld...

** Presubmit ERRORS **
tests/gclient_smoketest.py (54.86s) failed
Ffatal: reference is not a tree: a75cc338804c5e1bec343e2d92e13481e434ff8f
EE..............................................
======================================================================
ERROR: testBlinkDEPSChangeUsingGit (__main__.BlinkDEPSTransitionSmokeTest)
Like testBlinkDEPSChangeUsingGclient, but move the main project using
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1630, in testBlinkDEPSChangeUsingGit
    cwd=self.checkout_path)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
484, in check_call
    check_call_out(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
478, in check_call_out
    returncode, args, kwargs.get('cwd'), out[0], out[1])
CalledProcessError: Command git checkout -q
a75cc338804c5e1bec343e2d92e13481e434ff8f returned non-zero exit status 128 in
/tmp/trialMb_yOi/__main__.BlinkDEPSTransitionSmokeTest.testBlinkDEPSChangeUsingGit/src

======================================================================
ERROR: testBlinkLocalBranchesArePreserved
(__main__.BlinkDEPSTransitionSmokeTest)
Checks that the state of local git branches are effectively preserved
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1656, in
testBlinkLocalBranchesArePreserved
    self.CheckStatusPreMergePoint()
  File "tests/gclient_smoketest.py", line 1558, in CheckStatusPreMergePoint
    self.blink), self.blink_git_url)
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/scm.py", line
121, in Capture
    cwd=cwd, stderr=subprocess2.PIPE, env=env, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
515, in check_output
    return check_call_out(args, stdout=PIPE, **kwargs)[0]
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
475, in check_call_out
    out, returncode = communicate(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
449, in communicate
    proc = Popen(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
253, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory:
'/tmp/trialMb_yOi/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit'.
Check that
/tmp/trialMb_yOi/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit
or git exist and have execution permission.

======================================================================
FAIL: testBlinkDEPSChangeUsingGclient (__main__.BlinkDEPSTransitionSmokeTest)
Checks that {src,blink} repos are consistent when syncing going back and
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1601, in
testBlinkDEPSChangeUsingGclient
    self.assertEqual(res[2], 0, 'DEPS change sync failed.')
AssertionError: DEPS change sync failed.

----------------------------------------------------------------------
Ran 49 tests in 54.688s

FAILED (failures=1, errors=2)


Failed to checkout rietveld. Do you have mercurial installed?
Command hg clone -q -u 9349cab9a3bb -r 9349cab9a3bb
https://code.google.com/p/rietveld/
/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/_rietveld
returned non-zero exit status 255

Presubmit checks took 289.4s to calculate.

[pasko-google - do not use, 2015-04-03 14:16:30]

Cherry-picked it at: https://codereview.chromium.org/1059723003/

(presubmit failed for me asking to install python-coverage (3.7) and
google_appengine sdk, I believe these failures are unrelated to this patch)