Add URL recipe module from "depot_tools".

Add "url" recipe module to base.

Add the "url" recipe module, originally from "depot_tools", to the base
recipe modules. This module has a shiny new API, stronger tests, and
a more efficient and correct backend "pycurl.py" resource.

This also uses "vpython" to run URL commands in a VirtualEnv containing all
of the prerequisites for "requests" to perform secure HTTPS transactions.

BUG=None
TEST=local

Review-Url: https://codereview.chromium.org/2868333004
Committed: https://github.com/luci/recipes-py/commit/c6560f3627af81c9004024079525ef63a223d33a

[dnj (Google), 2017-05-10 22:38:05]

dnj@google.com changed reviewers:
+ dnj@google.com, iannucci@chromium.org

[dnj (Google), 2017-05-10 22:38:05]

PTAL

[iannucci, 2017-05-12 00:53:51]

iannucci@chromium.org changed reviewers:
+ phosek@chromium.org

[iannucci, 2017-05-12 00:53:52]

yay! lgtm

+phosek fyi

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api.py
File recipe_modules/url/api.py (right):

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:10: quote = staticmethod(urllib.quote)
buh? staticmethod shouldn't be needed

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:17: class Response(object):
note: capture error body and stuff in different field

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:26: def method(self):
DOOOOGGGGG STRIIING

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:35: raise ValueError('HTTP status (%d)' %
(self.status_code,))
real exception?

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:112: strip_prefix=None, log=False, **kwargs):
no kwargs just passthrough timeout

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
File recipe_modules/url/resources/pycurl.py (right):

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
recipe_modules/url/resources/pycurl.py:37: def write(self, content):
assert len(prefix) < CHUNK_SIZE, use iter() interface

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
recipe_modules/url/resources/pycurl.py:96: parser.add_argument('url', help='the
url to fetch')
urlparse validate (up in the recipe)?

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
recipe_modules/url/resources/pycurl.py:97: parser.add_argument('--status-json',
metavar='PATH', action='store',
required=True ?

[dnj, 2017-05-12 02:15:05]

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api.py
File recipe_modules/url/api.py (right):

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:10: quote = staticmethod(urllib.quote)
On 2017/05/12 00:53:52, iannucci wrote:
> buh? staticmethod shouldn't be needed

Done.

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:17: class Response(object):
On 2017/05/12 00:53:52, iannucci wrote:
> note: capture error body and stuff in different field

Done.

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:26: def method(self):
On 2017/05/12 00:53:52, iannucci wrote:
> DOOOOGGGGG STRIIING

Done.

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:35: raise ValueError('HTTP status (%d)' %
(self.status_code,))
On 2017/05/12 00:53:52, iannucci wrote:
> real exception?

Done.

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/api....
recipe_modules/url/api.py:112: strip_prefix=None, log=False, **kwargs):
On 2017/05/12 00:53:52, iannucci wrote:
> no kwargs just passthrough timeout

Done.

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
File recipe_modules/url/resources/pycurl.py (right):

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
recipe_modules/url/resources/pycurl.py:37: def write(self, content):
On 2017/05/12 00:53:52, iannucci wrote:
> assert len(prefix) < CHUNK_SIZE, use iter() interface

Done.

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
recipe_modules/url/resources/pycurl.py:96: parser.add_argument('url', help='the
url to fetch')
On 2017/05/12 00:53:52, iannucci wrote:
> urlparse validate (up in the recipe)?

Done.

https://codereview.chromium.org/2868333004/diff/40001/recipe_modules/url/reso...
recipe_modules/url/resources/pycurl.py:97: parser.add_argument('--status-json',
metavar='PATH', action='store',
On 2017/05/12 00:53:52, iannucci wrote:
> required=True ?

Done.

[iannucci, 2017-05-12 02:15:22]

The CQ bit was checked by iannucci@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-05-12 02:15:30]

Dry run: CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-05-12 02:18:39]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-05-12 02:18:40]

Dry run: This issue passed the CQ dry run.

[phosek, 2017-05-12 02:40:28]

phosek@google.com changed reviewers:
+ phosek@google.com

[phosek, 2017-05-12 02:40:29]

lgtm! awesome, waiting impatiently for this to land

[dnj, 2017-05-12 02:57:41]

On 2017/05/12 02:40:29, phosek wrote:
> lgtm! awesome, waiting impatiently for this to land

Sorry - blocked on a Kitchen production roll for the precursor "vpython" CL,
need sufficient canary runs before we can confidently make that push :(

[iannucci, 2017-05-12 07:52:59]

lgtm w/ nit

https://codereview.chromium.org/2868333004/diff/80001/recipe_modules/url/api.py
File recipe_modules/url/api.py (right):

https://codereview.chromium.org/2868333004/diff/80001/recipe_modules/url/api....
recipe_modules/url/api.py:189: headers = headers or {}
not needed? just move the sekuritay check into the if headers: block below?

[dnj, 2017-05-12 15:50:26]

https://codereview.chromium.org/2868333004/diff/80001/recipe_modules/url/api.py
File recipe_modules/url/api.py (right):

https://codereview.chromium.org/2868333004/diff/80001/recipe_modules/url/api....
recipe_modules/url/api.py:189: headers = headers or {}
On 2017/05/12 07:52:58, iannucci wrote:
> not needed? just move the sekuritay check into the if headers: block below?

Done.

[dnj, 2017-05-12 15:50:37]

Description was changed from

==========
Add URL recipe module from "depot_tools".

Add the "url" recipe module from "depot_tools". Rather than vendor in
"requests" and associates, we use the new "venv" Python execution
feature to pull it in via inline "vpython".

BUG=None
TEST=local
==========

to

==========
Add "url" recipe module to base.

Add the "url" recipe module, originally from "depot_tools", to the base
recipe modules. This module has a shiny new API, stronger tests, and
a more efficient and correct backend "pycurl.py" resource.

This also uses "vpython" to run URL commands in a VirtualEnv containing all
of the prerequisites for "requests" to perform secure HTTPS transactions.

BUG=None
TEST=local
==========

[dnj, 2017-05-12 15:54:42]

Update comments with exceptions

[dnj, 2017-05-12 16:01:34]

The CQ bit was checked by dnj@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-05-12 16:01:35]

Dry run: CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-05-12 16:04:18]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-05-12 16:04:19]

Dry run: This issue passed the CQ dry run.

[dnj, 2017-05-12 16:04:32]

The CQ bit was checked by dnj@chromium.org

[dnj, 2017-05-12 16:04:32]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org, phosek@google.com
Link to the patchset: https://codereview.chromium.org/2868333004/#ps140001
(title: "Update comments with exceptions")

[commit-bot: I haz the power, 2017-05-12 16:04:40]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-05-12 16:07:31]

CQ is committing da patch.

Bot data: {"patchset_id": 140001, "attempt_start_ts": 1494605072530220,
"parent_rev": "60cc7f3f49737e8cf852e396b4436c2b6bd3f56d", "commit_rev":
"c6560f3627af81c9004024079525ef63a223d33a"}

[commit-bot: I haz the power, 2017-05-12 16:07:36]

Description was changed from

==========
Add "url" recipe module to base.

Add the "url" recipe module, originally from "depot_tools", to the base
recipe modules. This module has a shiny new API, stronger tests, and
a more efficient and correct backend "pycurl.py" resource.

This also uses "vpython" to run URL commands in a VirtualEnv containing all
of the prerequisites for "requests" to perform secure HTTPS transactions.

BUG=None
TEST=local
==========

to

==========
Add "url" recipe module to base.

Add the "url" recipe module, originally from "depot_tools", to the base
recipe modules. This module has a shiny new API, stronger tests, and
a more efficient and correct backend "pycurl.py" resource.

This also uses "vpython" to run URL commands in a VirtualEnv containing all
of the prerequisites for "requests" to perform secure HTTPS transactions.

BUG=None
TEST=local

Review-Url: https://codereview.chromium.org/2868333004
Committed:
https://github.com/luci/recipes-py/commit/c6560f3627af81c9004024079525ef63a22...
==========

[commit-bot: I haz the power, 2017-05-12 16:07:37]

Committed patchset #8 (id:140001) as
https://github.com/luci/recipes-py/commit/c6560f3627af81c9004024079525ef63a22...