OAuth2 support in depot_tools

Added OAuth2 support in depot_tools

BUG=356813
R=nodir@chromium.org, vadimsh@chromium.org

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=294698

[sheyang, 2015-03-05 00:59:51]

sheyang@chromium.org changed reviewers:
+ nodir@chromium.org

[sheyang, 2015-03-05 01:09:12]

First draft of git cl try...

[nodir, 2015-03-05 03:58:56]

https://codereview.chromium.org/963953003/diff/40001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode247
git_cl.py:247: def trigger_distributed_try_jobs(
what distributed means in this context?

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode263
git_cl.py:263: bb_master = _prefix_master(master)
call it bucket

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode292
git_cl.py:292: req.execute()
You ignore the result that may contain an error. I'd check and report it.

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode2883
git_cl.py:2883: trigger_distributed_try_jobs(creds, cl, options, masters, 'git
cl try')
Please do not commit this CL until we move/replicate try job access list to
chrome-infra-auth, otherwise git-cl-try will stop working for many

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode2889
git_cl.py:2889: print 'Tryjob request failed: %s.' % e
typo: ex

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py
File git_cl_oauth2.py (right):

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode6
git_cl_oauth2.py:6: """OAuth2 related utilities and implementation got git cl
commands."""
what "got" means here? Did you mean "for"?

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode16
git_cl_oauth2.py:16: from third_party.google_api_python_client import apiclient
sort imports

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode23
git_cl_oauth2.py:23: 'client_secret': 'DddcCK1d6_ADwxqGDEGlsisy'
Make two constants instead of one dict

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode28
git_cl_oauth2.py:28: storage_dir = os.path.expanduser('~/git_cl_credentials')
Does this work on windows? I'd use os.path.expanduser(os.path.join('~',
'.git_cl_credentials') instead

Also make the directory private: prefix with .

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode31
git_cl_oauth2.py:31: storage_path = os.path.join(storage_dir, '.%s' %
code_review_server)
I'd make the parent dir invisible, and the files visible

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode34
git_cl_oauth2.py:34: 
nit: double blank lines between top-level definitions everywhere

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode36
git_cl_oauth2.py:36: print 'Fetching OAuth2 credential from local storage ...'
Please use plural form "credentials", "creds", everywhere in this file
(docstrings, vars, funcs)

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode36
git_cl_oauth2.py:36: print 'Fetching OAuth2 credential from local storage ...'
Use logging.info or logging.debug instead of print in this file. This one should
be probably debug

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode55
git_cl_oauth2.py:55: '--auth-host-port',
use either - or _ between words in options, be consistent. I think chrome-infra
standard is dashes

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode57
git_cl_oauth2.py:57: action="append",
Use single quotes everywhere

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode83
git_cl_oauth2.py:83: """Get OAuth2 credential."""
Please document args. Is code_review_server a hostname or URL with a schema? If
former, I'dcall it code_review_hostname

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode93
git_cl_oauth2.py:93: scope='email',
please make it a constant

[sheyang, 2015-03-06 00:29:11]

https://codereview.chromium.org/963953003/diff/40001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode247
git_cl.py:247: def trigger_distributed_try_jobs(
On 2015/03/05 03:58:55, nodir wrote:
> what distributed means in this context?

Just keep the same name. 

I guess the original meaning is about the ability to schedule tryjobs across
masters.

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode263
git_cl.py:263: bb_master = _prefix_master(master)
On 2015/03/05 03:58:55, nodir wrote:
> call it bucket

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode2883
git_cl.py:2883: trigger_distributed_try_jobs(creds, cl, options, masters, 'git
cl try')
On 2015/03/05 03:58:55, nodir wrote:
> Please do not commit this CL until we move/replicate try job access list to
> chrome-infra-auth, otherwise git-cl-try will stop working for many

I won't - see COMMIT=false in the description.

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode2889
git_cl.py:2889: print 'Tryjob request failed: %s.' % e
On 2015/03/05 03:58:55, nodir wrote:
> typo: ex

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py
File git_cl_oauth2.py (right):

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode6
git_cl_oauth2.py:6: """OAuth2 related utilities and implementation got git cl
commands."""
On 2015/03/05 03:58:56, nodir wrote:
> what "got" means here? Did you mean "for"?

Yep... Fixed.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode16
git_cl_oauth2.py:16: from third_party.google_api_python_client import apiclient
On 2015/03/05 03:58:55, nodir wrote:
> sort imports

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode23
git_cl_oauth2.py:23: 'client_secret': 'DddcCK1d6_ADwxqGDEGlsisy'
On 2015/03/05 03:58:56, nodir wrote:
> Make two constants instead of one dict

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode28
git_cl_oauth2.py:28: storage_dir = os.path.expanduser('~/git_cl_credentials')
On 2015/03/05 03:58:56, nodir wrote:
> Does this work on windows? I'd use os.path.expanduser(os.path.join('~',
> '.git_cl_credentials') instead
> 
> Also make the directory private: prefix with .

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode31
git_cl_oauth2.py:31: storage_path = os.path.join(storage_dir, '.%s' %
code_review_server)
On 2015/03/05 03:58:55, nodir wrote:
> I'd make the parent dir invisible, and the files visible

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode34
git_cl_oauth2.py:34: 
On 2015/03/05 03:58:55, nodir wrote:
> nit: double blank lines between top-level definitions everywhere

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode36
git_cl_oauth2.py:36: print 'Fetching OAuth2 credential from local storage ...'
On 2015/03/05 03:58:56, nodir wrote:
> Please use plural form "credentials", "creds", everywhere in this file
> (docstrings, vars, funcs)

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode36
git_cl_oauth2.py:36: print 'Fetching OAuth2 credential from local storage ...'
On 2015/03/05 03:58:55, nodir wrote:
> Use logging.info or logging.debug instead of print in this file. This one
should
> be probably debug

Done.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode55
git_cl_oauth2.py:55: '--auth-host-port',
On 2015/03/05 03:58:56, nodir wrote:
> use either - or _ between words in options, be consistent. I think
chrome-infra
> standard is dashes

use - now.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode57
git_cl_oauth2.py:57: action="append",
On 2015/03/05 03:58:56, nodir wrote:
> Use single quotes everywhere

Copy-Paste error... Fixed.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode83
git_cl_oauth2.py:83: """Get OAuth2 credential."""
On 2015/03/05 03:58:56, nodir wrote:
> Please document args. Is code_review_server a hostname or URL with a schema?
If
> former, I'dcall it code_review_hostname

CODE_REVIEW_SERVER is the same key name used in codereview.settings, so I'd
prefer to leave it as is.

Added docs.

https://codereview.chromium.org/963953003/diff/40001/git_cl_oauth2.py#newcode93
git_cl_oauth2.py:93: scope='email',
On 2015/03/05 03:58:56, nodir wrote:
> please make it a constant

Done.

[nodir, 2015-03-10 22:39:40]

https://codereview.chromium.org/963953003/diff/40001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/40001/git_cl.py#newcode2883
git_cl.py:2883: trigger_distributed_try_jobs(creds, cl, options, masters, 'git
cl try')
On 2015/03/06 00:29:10, sheyang wrote:
> On 2015/03/05 03:58:55, nodir wrote:
> > Please do not commit this CL until we move/replicate try job access list to
> > chrome-infra-auth, otherwise git-cl-try will stop working for many
> 
> I won't - see COMMIT=false in the description.

Acknowledged.

https://codereview.chromium.org/963953003/diff/60001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode252
git_cl.py:252: requester = creds_props['id_token']['email']
I think you can get requester like this: credentials.id_token.email

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode297
git_cl.py:297: response = req.execute()
do try_count -=1 after this

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode303
git_cl.py:303: try_count = 0
If this intends to break the loop, please do `break` instead

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode311
git_cl.py:311: try_count -= 1
not needed if try_count -=1 right after trying

https://codereview.chromium.org/963953003/diff/60001/git_cl_oauth2.py
File git_cl_oauth2.py (right):

https://codereview.chromium.org/963953003/diff/60001/git_cl_oauth2.py#newcode13
git_cl_oauth2.py:13: from third_party.google_api_python_client import apiclient
this is still not sorted

https://codereview.chromium.org/963953003/diff/60001/git_cl_oauth2.py#newcode58
git_cl_oauth2.py:58: action="append",
this still uses double quotes

[sheyang, 2015-03-13 20:58:28]

https://codereview.chromium.org/963953003/diff/60001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode252
git_cl.py:252: requester = creds_props['id_token']['email']
On 2015/03/10 22:39:40, nodir wrote:
> I think you can get requester like this: credentials.id_token.email

No it doesn't work.

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode297
git_cl.py:297: response = req.execute()
On 2015/03/10 22:39:40, nodir wrote:
> do try_count -=1 after this

You mean before this right? Otherwise we may never hit this line if
req.execute() throws a transient error.

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode303
git_cl.py:303: try_count = 0
On 2015/03/10 22:39:40, nodir wrote:
> If this intends to break the loop, please do `break` instead

Done.

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode311
git_cl.py:311: try_count -= 1
On 2015/03/10 22:39:40, nodir wrote:
> not needed if try_count -=1 right after trying

Done.

https://codereview.chromium.org/963953003/diff/60001/git_cl_oauth2.py
File git_cl_oauth2.py (right):

https://codereview.chromium.org/963953003/diff/60001/git_cl_oauth2.py#newcode13
git_cl_oauth2.py:13: from third_party.google_api_python_client import apiclient
On 2015/03/10 22:39:40, nodir wrote:
> this is still not sorted

Don;t remember the rule to sort 'from...import...'. Try to fix it...

https://codereview.chromium.org/963953003/diff/60001/git_cl_oauth2.py#newcode58
git_cl_oauth2.py:58: action="append",
On 2015/03/10 22:39:40, nodir wrote:
> this still uses double quotes

Done.

[nodir, 2015-03-18 23:49:16]

lgtm

https://codereview.chromium.org/963953003/diff/60001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/60001/git_cl.py#newcode297
git_cl.py:297: response = req.execute()
On 2015/03/13 20:58:27, sheyang wrote:
> On 2015/03/10 22:39:40, nodir wrote:
> > do try_count -=1 after this
> 
> You mean before this right? Otherwise we may never hit this line if
> req.execute() throws a transient error.

Right

https://codereview.chromium.org/963953003/diff/100001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/100001/git_cl.py#newcode2942
git_cl.py:2942: print 'Unexcpected error when trying to trigger tryjobs: %s.' %
e
typo: unexpected

[sheyang, 2015-03-24 00:36:30]

https://codereview.chromium.org/963953003/diff/100001/git_cl.py
File git_cl.py (right):

https://codereview.chromium.org/963953003/diff/100001/git_cl.py#newcode2942
git_cl.py:2942: print 'Unexcpected error when trying to trigger tryjobs: %s.' %
e
On 2015/03/18 23:49:15, nodir wrote:
> typo: unexpected

Done.

[Vadim Sh., 2015-04-07 00:42:14]

vadimsh@chromium.org changed reviewers:
+ vadimsh@chromium.org

[Vadim Sh., 2015-04-07 00:42:15]

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py
File git_cl_oauth2.py (right):

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py#newcode1
git_cl_oauth2.py:1: #!/usr/bin/env python
this file is not executable, there's no __main__ here

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py#newcode6
git_cl_oauth2.py:6: """OAuth2 related utilities and implementation for git cl
commands."""
Can you please rename this file to "oauth2.py" since there's nothing git cl
specific here.

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py#newcode13
git_cl_oauth2.py:13: from third_party.google_api_python_client import apiclient
This is not used in this file.

[sheyang, 2015-04-07 02:09:27]

Updated description and addressed comments.

PTAL.

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py
File git_cl_oauth2.py (right):

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py#newcode1
git_cl_oauth2.py:1: #!/usr/bin/env python
On 2015/04/07 00:42:14, Vadim Sh. wrote:
> this file is not executable, there's no __main__ here

Removed this line since this file is not meant to be executable.

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py#newcode6
git_cl_oauth2.py:6: """OAuth2 related utilities and implementation for git cl
commands."""
On 2015/04/07 00:42:14, Vadim Sh. wrote:
> Can you please rename this file to "oauth2.py" since there's nothing git cl
> specific here.

Done.

https://codereview.chromium.org/963953003/diff/120001/git_cl_oauth2.py#newcode13
git_cl_oauth2.py:13: from third_party.google_api_python_client import apiclient
On 2015/04/07 00:42:14, Vadim Sh. wrote:
> This is not used in this file.

Done.

[Vadim Sh., 2015-04-07 02:15:37]

lgtm

[sheyang, 2015-04-07 02:19:35]

The CQ bit was checked by sheyang@chromium.org

[sheyang, 2015-04-07 02:19:36]

The patchset sent to the CQ was uploaded after l-g-t-m from nodir@chromium.org
Link to the patchset: https://codereview.chromium.org/963953003/#ps140001
(title: "restore git_cl")

[commit-bot: I haz the power, 2015-04-07 02:19:46]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/963953003/140001

[commit-bot: I haz the power, 2015-04-07 02:20:00]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-07 02:20:01]

Failed to apply patch for depot_tools/oauth2.py:
While running svn add depot_tools/oauth2.py --force --config-dir
/b/infra_internal/commit_queue/subversion_config --non-interactive;
  patching file oauth2.py
  svn: warning: W155010:
'/b/infra_internal/commit_queue/workdir/tools/depot_tools/oauth2.py' not found
  svn: E200009: Could not add all targets because some targets don't exist
  svn: E200009: Illegal target for the requested operation

Patch:   N   depot_tools/oauth2.py
Index: oauth2.py
diff --git depot_tools/oauth2.py depot_tools/oauth2.py
new file mode 100644
index
0000000000000000000000000000000000000000..08f0abf29c23f5da356899c0419587e2795b58e8
--- /dev/null
+++ b/depot_tools/oauth2.py
@@ -0,0 +1,103 @@
+# Copyright (c) 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""OAuth2 related utilities and implementation for git cl commands."""
+
+import copy
+import logging
+import optparse
+import os
+
+from third_party.oauth2client import tools
+from third_party.oauth2client.file import Storage
+import third_party.oauth2client.client as oa2client
+
+
+REDIRECT_URI = 'urn:ietf:wg:oauth:2.0:oob'
+CLIENT_ID = ('174799409470-8k3b89iov4racu9jrf7if3k4591voig3'
+             '.apps.googleusercontent.com')
+CLIENT_SECRET = 'DddcCK1d6_ADwxqGDEGlsisy'
+SCOPE = 'email'
+
+
+def _fetch_storage(code_review_server):
+  storage_dir = os.path.expanduser(os.path.join('~', '.git_cl_credentials'))
+  if not os.path.isdir(storage_dir):
+    os.makedirs(storage_dir)
+  storage_path = os.path.join(storage_dir, code_review_server)
+  storage = Storage(storage_path)
+  return storage
+
+
+def _fetch_creds_from_storage(storage):
+  logging.debug('Fetching OAuth2 credentials from local storage ...')
+  credentials = storage.get()
+  if not credentials or credentials.invalid:
+    return None
+  if not credentials.access_token or credentials.access_token_expired:
+    return None
+  return credentials
+
+
+def add_oauth2_options(parser):
+  """Add OAuth2-related options."""
+  group = optparse.OptionGroup(parser, "OAuth2 options")
+  group.add_option(
+      '--auth-host-name',
+      default='localhost',
+      help='Host name to use when running a local web server '
+           'to handle redirects during OAuth authorization.'
+           'Default: localhost.'
+  )
+  group.add_option(
+      '--auth-host-port',
+      type=int,
+      action='append',
+      default=[8080, 8090],
+      help='Port to use when running a local web server to handle '
+           'redirects during OAuth authorization. '
+           'Repeat this option to specify a list of values.'
+           'Default: [8080, 8090].'
+  )
+  group.add_option(
+      '--noauth-local-webserver',
+      action='store_true',
+      default=False,
+      help='Run a local web server to handle redirects '
+           'during OAuth authorization.'
+           'Default: False.'
+  )
+  group.add_option(
+      '--no-cache',
+      action='store_true',
+      default=False,
+      help='Get fresh credentials from web server instead of using '
+           'the crendentials stored on a local storage file.'
+           'Default: False.'
+  )
+  parser.add_option_group(group)
+
+
+def get_oauth2_creds(options, code_review_server):
+  """Get OAuth2 credentials.
+
+  Args:
+    options: Command line options.
+    code_review_server: Code review server name, e.g., codereview.chromium.org.
+  """
+  storage = _fetch_storage(code_review_server)
+  creds = None
+  if not options.no_cache:
+    creds = _fetch_creds_from_storage(storage)
+  if creds is None:
+    logging.debug('Fetching OAuth2 credentials from web server...')
+    flow = oa2client.OAuth2WebServerFlow(
+        client_id=CLIENT_ID,
+        client_secret=CLIENT_SECRET,
+        scope=SCOPE,
+        redirect_uri=REDIRECT_URI)
+    flags = copy.deepcopy(options)
+    flags.logging_level = 'WARNING'
+    creds = tools.run_flow(flow, storage, flags)
+  return creds

[sheyang, 2015-04-07 02:23:38]

Committed patchset #8 (id:140001) manually as 294698 (presubmit successful).