Add OAuth2 support for end users (i.e. 3-legged flow with the browser).

Add OAuth2 support for end users (i.e. 3-legged flow with the browser).

This CL introduces new top level command for managing cached auth tokens:
  $ depot-tools-auth login codereview.chromium.org
  $ depot-tools-auth info codereview.chromium.org
  $ depot-tools-auth logout codereview.chromium.org

All scripts that use rietveld.Rietveld internally should be able to use cached
credentials created by 'depot-tools-auth' subcommand. Also 'depot-tools-auth'
is the only way to run login flow. If some scripts stumbles over expired or
revoked token, it dies with the error, asking user to run
'depot-tools-auth login <hostname>'.

Password login is still default. OAuth2 can be enabled by passing --oauth2 to
all scripts.

R=maruel@chromium.org
BUG=356813
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=294764

[Vadim Sh., 2015-04-09 06:21:59]

PTAL for general approach. 

I'll finish this CL tomorrow.

Uploaded with --oauth2 :) Server side seems to be working fine (I asked Robbie
yesterday to add the client_id to list of accepted clients on
codereview.chromium.org).

https://codereview.chromium.org/1074673002/diff/1/auth.py
File auth.py (right):

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode40
auth.py:40: OAUTH_CLIENT_SECRET = 'uBfbay2KCy9t4QveJ-dOqHtp'
same ones that in Go code. Application title is "Chrome Infra". Probably can be
changed to something else? https://screenshot.googleplex.com/OjSRZoVZaq.png

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode49
auth.py:49: os.path.expanduser('~'), '.depot_tools_oauth2_tokens')
by analogy with ~/.appcfg_oauth2_tokens that appcfg.py is using.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode78
auth.py:78: # HACK(vadimsh): It is assumed here that the token cache key is a
hostname.
this hostname business is annoying, but needed to support tokens for public and
internal rietveld with different emails

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode232
auth.py:232: def has_cached_credentials(self):
I plan to use it to die fast in 'git cl upload' if there's no cached creds. Not
plugged it in yet.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode293
auth.py:293: def authorize(self, http):
Google APIs Client library likes Http2. This method allows to reuse same exact
authentication logic with Google APIs Client library by implementing integration
with Http2.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode335
auth.py:335: return multistore_file.get_credential_storage_custom_string_key(
I vaguely remember mutlistore_file to blow up on Windows. Not sure... Will ask
volunteers on chromium-dev to start using oauth2 before enabling it by default
to tests better.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode367
auth.py:367: credentials = None
will eventually add service accounts support here reusing code from swarming
client (that doesn't need OpenSSL, yay)

https://codereview.chromium.org/1074673002/diff/1/third_party/oauth2client/lo...
File third_party/oauth2client/locked_file.py (right):

https://codereview.chromium.org/1074673002/diff/1/third_party/oauth2client/lo...
third_party/oauth2client/locked_file.py:38: from . import util
will update MODIFICATIONS.diff later (it already exists and already contains
similar import hacks).

[M-A Ruel, 2015-04-09 20:00:40]

https://codereview.chromium.org/1074673002/diff/1/auth.py
File auth.py (right):

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode40
auth.py:40: OAUTH_CLIENT_SECRET = 'uBfbay2KCy9t4QveJ-dOqHtp'
On 2015/04/09 06:21:59, Vadim Sh. wrote:
> same ones that in Go code. Application title is "Chrome Infra". Probably can
be
> changed to something else? https://screenshot.googleplex.com/OjSRZoVZaq.png

No, it's fine.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode49
auth.py:49: os.path.expanduser('~'), '.depot_tools_oauth2_tokens')
On 2015/04/09 06:21:59, Vadim Sh. wrote:
> by analogy with ~/.appcfg_oauth2_tokens that appcfg.py is using.

I'd prefer to put stuff in ~/.config/ on posix but oh well; my ~ is littered
with crap, we could do our thing to help the situation. :)

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode102
auth.py:102: default(use_local_webserver, True),
Can you use instead:
bool(sys.platform != 'linux' or os.environ.get('DISPLAY'))

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode129
auth.py:129: '--auth-tokens-cache',
Why make this configurable?

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode183
auth.py:183: hostname = hostname.lower()
not a fan of doing a .lower() here. It should abort on non-lower case instead.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode187
auth.py:187: return parsed.netloc
It should abort if there's any .path, .params, .query or .fragment.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode286
auth.py:286: uri=(
Extraneous () is not needed here

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode335
auth.py:335: return multistore_file.get_credential_storage_custom_string_key(
On 2015/04/09 06:21:59, Vadim Sh. wrote:
> I vaguely remember mutlistore_file to blow up on Windows. Not sure... Will ask
> volunteers on chromium-dev to start using oauth2 before enabling it by default
> to tests better.

I still have a Windows box, I can test on it.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode377
auth.py:377: logging.error(
warning?

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py
File depot-tools-auth.py (right):

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py#newcode6
depot-tools-auth.py:6: """Tool that manages OAuth2 token cache used by other
depot_tools scripts.
"""Manages ...

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py#newcode28
depot-tools-auth.py:28: """Perform interactive login and caches authentication
token."""
Performs

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py#newcode34
depot-tools-auth.py:34: 
I'd prefer if you return 0 explicitly everywhere.

[Vadim Sh., 2015-04-10 01:18:43]

PTAL

This CL is now complete (also rebased).

USE_OAUTH2 file can be put into depot_tools/ directory to enable OAuth2 by
default before it will be a forced default. I choose it over environment
variable, because adding an env variable that sticks across shells is PITA.
Though if you insist on environment variable, I can use it instead.

Once landed, I plan to write a man page (if figure out how) for
depot-tools-auth, test is on Windows, and then send PSA to chromium-dev asking
brave souls to "touch depot_tools/USE_OAUTH2".

https://codereview.chromium.org/1074673002/diff/1/auth.py
File auth.py (right):

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode49
auth.py:49: os.path.expanduser('~'), '.depot_tools_oauth2_tokens')
On 2015/04/09 20:00:40, M-A Ruel wrote:
> On 2015/04/09 06:21:59, Vadim Sh. wrote:
> > by analogy with ~/.appcfg_oauth2_tokens that appcfg.py is using.
> 
> I'd prefer to put stuff in ~/.config/ on posix but oh well; my ~ is littered
> with crap, we could do our thing to help the situation. :)

In presence of multiple platforms (Win, Mac), it is simpler to put crap in '~'
directly. E.g. if we ever want to deploy this file automatically (ala .netrc),
consistent location across all platforms is important.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode102
auth.py:102: default(use_local_webserver, True),
On 2015/04/09 20:00:40, M-A Ruel wrote:
> Can you use instead:
> bool(sys.platform != 'linux' or os.environ.get('DISPLAY')) 

Done.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode129
auth.py:129: '--auth-tokens-cache',
On 2015/04/09 20:00:40, M-A Ruel wrote:
> Why make this configurable?

Some folks wanted it for swarming.py for some reason.

Removed. With depot_tools it is easier to add stuff later, rather than remove it
once added.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode183
auth.py:183: hostname = hostname.lower()
On 2015/04/09 20:00:40, M-A Ruel wrote:
> not a fan of doing a .lower() here. It should abort on non-lower case instead.

1) DNS names are case insensitive.
2) Some codereview.settings files in the wild use uppercase. 
3) I don't want to scan all depot_tools code to verify all domain names passed
to this functions are converted to lower case first.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode187
auth.py:187: return parsed.netloc
On 2015/04/09 20:00:40, M-A Ruel wrote:
> It should abort if there's any .path, .params, .query or .fragment.

Done.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode286
auth.py:286: uri=(
On 2015/04/09 20:00:40, M-A Ruel wrote:
> Extraneous () is not needed here

Done.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode335
auth.py:335: return multistore_file.get_credential_storage_custom_string_key(
On 2015/04/09 20:00:40, M-A Ruel wrote:
> On 2015/04/09 06:21:59, Vadim Sh. wrote:
> > I vaguely remember mutlistore_file to blow up on Windows. Not sure... Will
ask
> > volunteers on chromium-dev to start using oauth2 before enabling it by
default
> > to tests better.
> 
> I still have a Windows box, I can test on it.

Acknowledged.

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode377
auth.py:377: logging.error(
On 2015/04/09 20:00:40, M-A Ruel wrote:
> warning?

Done.

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py
File depot-tools-auth.py (right):

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py#newcode6
depot-tools-auth.py:6: """Tool that manages OAuth2 token cache used by other
depot_tools scripts.
On 2015/04/09 20:00:40, M-A Ruel wrote:
> """Manages ...

Done.

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py#newcode28
depot-tools-auth.py:28: """Perform interactive login and caches authentication
token."""
On 2015/04/09 20:00:40, M-A Ruel wrote:
> Performs

Done.

https://codereview.chromium.org/1074673002/diff/1/depot-tools-auth.py#newcode34
depot-tools-auth.py:34: 
On 2015/04/09 20:00:40, M-A Ruel wrote:
> I'd prefer if you return 0 explicitly everywhere.

Done.

[M-A Ruel, 2015-04-10 01:25:56]

lgtm

https://codereview.chromium.org/1074673002/diff/1/auth.py
File auth.py (right):

https://codereview.chromium.org/1074673002/diff/1/auth.py#newcode129
auth.py:129: '--auth-tokens-cache',
On 2015/04/10 01:18:42, Vadim Sh. wrote:
> On 2015/04/09 20:00:40, M-A Ruel wrote:
> > Why make this configurable?
> 
> Some folks wanted it for swarming.py for some reason.

Me? I don't recall. I don't think it's used anyway.

> Removed. With depot_tools it is easier to add stuff later, rather than remove
it
> once added.

https://codereview.chromium.org/1074673002/diff/20001/depot-tools-auth.bat
File depot-tools-auth.bat (right):

https://codereview.chromium.org/1074673002/diff/20001/depot-tools-auth.bat#ne...
depot-tools-auth.bat:2: :: Copyright (c) 2015 The Chromium Authors. All rights
reserved.
The 2015 flavor is to not use (c) anymore.

[Vadim Sh., 2015-04-10 02:11:56]

Also removed (unused) oauth2.py since it was completely superseded by auth.py in
functionality.

https://codereview.chromium.org/1074673002/diff/20001/depot-tools-auth.bat
File depot-tools-auth.bat (right):

https://codereview.chromium.org/1074673002/diff/20001/depot-tools-auth.bat#ne...
depot-tools-auth.bat:2: :: Copyright (c) 2015 The Chromium Authors. All rights
reserved.
On 2015/04/10 01:25:56, M-A Ruel wrote:
> The 2015 flavor is to not use (c) anymore.

Done.

[Vadim Sh., 2015-04-10 03:08:58]

I've tried to use OAuth2 with internal rietveld (that is not yet configured) and
found couple bugs.

PTAL at patch set 5.

https://codereview.chromium.org/1074673002/diff/60001/third_party/upload.py
File third_party/upload.py (right):

https://codereview.chromium.org/1074673002/diff/60001/third_party/upload.py#n...
third_party/upload.py:762: auth_func = KeyringCreds(server, host,
email).GetUserCredentials,
Stupid comma :) Didn't notice it when copy pasted the line from function
argument list.

[M-A Ruel, 2015-04-10 13:40:59]

lgtm

https://codereview.chromium.org/1074673002/diff/80001/.gitignore
File .gitignore (right):

https://codereview.chromium.org/1074673002/diff/80001/.gitignore#newcode50
.gitignore:50: # Ignore "flag" file used by auth.py.
This should eventually removed, so a TODO(vadimsh): Remove me would be good.

[Vadim Sh., 2015-04-10 18:45:27]

https://codereview.chromium.org/1074673002/diff/80001/.gitignore
File .gitignore (right):

https://codereview.chromium.org/1074673002/diff/80001/.gitignore#newcode50
.gitignore:50: # Ignore "flag" file used by auth.py.
On 2015/04/10 13:40:59, M-A Ruel wrote:
> This should eventually removed, so a TODO(vadimsh): Remove me would be good.

Done.

[Vadim Sh., 2015-04-10 21:18:51]

The CQ bit was checked by vadimsh@chromium.org

[Vadim Sh., 2015-04-10 21:18:52]

The patchset sent to the CQ was uploaded after l-g-t-m from maruel@chromium.org
Link to the patchset: https://codereview.chromium.org/1074673002/#ps100001
(title: " ")

[commit-bot: I haz the power, 2015-04-10 21:19:21]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1074673002/100001

[commit-bot: I haz the power, 2015-04-10 21:19:32]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-10 21:19:34]

Failed to apply patch for depot_tools/depot-tools-auth.bat:
While running patch -p1 --forward --force --no-backup-if-mismatch;
  A         depot_tools/depot-tools-auth.bat
  Copied depot_tools/gn.bat -> depot_tools/depot-tools-auth.bat
  patching file depot_tools/depot-tools-auth.bat
  Hunk #1 FAILED at 1 (different line endings).
  1 out of 1 hunk FAILED -- saving rejects to file
depot_tools/depot-tools-auth.bat.rej

Patch:   NR  depot_tools/gn.bat->depot_tools/depot-tools-auth.bat
Index: depot-tools-auth.bat
diff --git depot_tools/gn.bat depot_tools/depot-tools-auth.bat
similarity index 65%
copy from depot_tools/gn.bat
copy to depot_tools/depot-tools-auth.bat
index
250e4d9ac380c1baaa4f01a221c2f4c694de78e1..3e547f0cf083ff9e7139a336c65ff6bd7963c06d
100644
--- a/depot_tools/gn.bat
+++ b/depot_tools/depot-tools-auth.bat
@@ -1,11 +1,11 @@
-@echo off
-:: Copyright 2013 The Chromium Authors. All rights reserved.
-:: Use of this source code is governed by a BSD-style license that can be
-:: found in the LICENSE file.
-setlocal
-
-:: This is required with cygwin only.
-PATH=%~dp0;%PATH%
-
-:: Defer control.
-%~dp0python "%~dp0\gn.py" %*
+@echo off
+:: Copyright 2015 The Chromium Authors. All rights reserved.
+:: Use of this source code is governed by a BSD-style license that can be
+:: found in the LICENSE file.
+setlocal
+
+:: This is required with cygwin only.
+PATH=%~dp0;%PATH%
+
+:: Defer control.
+%~dp0python "%~dp0\depot-tools-auth.py" %*

[Vadim Sh., 2015-04-10 21:27:05]

The CQ bit was checked by vadimsh@chromium.org

[Vadim Sh., 2015-04-10 21:27:06]

The patchset sent to the CQ was uploaded after l-g-t-m from maruel@chromium.org
Link to the patchset: https://codereview.chromium.org/1074673002/#ps120001
(title: "windows lineendings for depot-tools-auth.bat")

[commit-bot: I haz the power, 2015-04-10 21:27:57]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1074673002/120001

[commit-bot: I haz the power, 2015-04-10 21:28:10]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-10 21:28:13]

Failed to apply patch for depot_tools/depot-tools-auth.bat:
While running patch -p1 --forward --force --no-backup-if-mismatch;
  A         depot_tools/depot-tools-auth.bat
  Copied depot_tools/gn.bat -> depot_tools/depot-tools-auth.bat
  patching file depot_tools/depot-tools-auth.bat
  Hunk #1 FAILED at 1 (different line endings).
  Hunk #2 FAILED at 8 (different line endings).
  2 out of 2 hunks FAILED -- saving rejects to file
depot_tools/depot-tools-auth.bat.rej

Patch:   NR  depot_tools/gn.bat->depot_tools/depot-tools-auth.bat
Index: depot-tools-auth.bat
diff --git depot_tools/gn.bat depot_tools/depot-tools-auth.bat
similarity index 63%
copy from depot_tools/gn.bat
copy to depot_tools/depot-tools-auth.bat
index
250e4d9ac380c1baaa4f01a221c2f4c694de78e1..fe13f9338be173613cdb857f70fa3920d03126f5
100644
--- a/depot_tools/gn.bat
+++ b/depot_tools/depot-tools-auth.bat
@@ -1,5 +1,5 @@
 @echo off
-:: Copyright 2013 The Chromium Authors. All rights reserved.
+:: Copyright 2015 The Chromium Authors. All rights reserved.
 :: Use of this source code is governed by a BSD-style license that can be
 :: found in the LICENSE file.
 setlocal
@@ -8,4 +8,4 @@ setlocal
 PATH=%~dp0;%PATH%
 
 :: Defer control.
-%~dp0python "%~dp0\gn.py" %*
+%~dp0python "%~dp0\depot-tools-auth.py" %*

[Vadim Sh., 2015-04-10 21:30:25]

Committed patchset #7 (id:120001) manually as 294764.