Add depot_tools builder to infra waterfall.

Add depot_tools builder to infra waterfall.

This builder clones, packs, and zips depot_tools for the purpose of
updating the windows depot_tools.zip bootstrap, and it also uploads the
generated `man/html` documentation folder to the chrome-infra-docs
bucket.

R=agable@chromium.org, dnj@chromium.org
BUG=598958
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299636

[dnj, 2016-04-01 14:40:01]

lgtm w/ nits

https://codereview.chromium.org/1849113003/diff/20001/scripts/slave/recipes/i...
File scripts/slave/recipes/infra/depot_tools_builder.py (right):

https://codereview.chromium.org/1849113003/diff/20001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/depot_tools_builder.py:42: api.git('clone',
'--single-branch', '-n', REPO_URL, api.path['checkout'])
nit: Maybe nested step here?

https://codereview.chromium.org/1849113003/diff/20001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/depot_tools_builder.py:48: api.git('reflog',
'expire', '--all')
Might be worth adding this sort of thing to the Git recipe module, "compact" or
something, b/c I can see it being useful elsewhere.

https://codereview.chromium.org/1849113003/diff/20001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/depot_tools_builder.py:59:
'gs://chrome-infra-docs/flat/depot_tools/docs/'],
nit: make this a constant up top

[agable, 2016-04-01 17:34:17]

Big top-level comment:
You're putting this on the chromium.infra waterfall, but it doesn't run any
tests. In my opinion, either:
a) This should be on chromium.infra.cron, since it does a thing that should be
done regularly; or
b) This should be on chromium.infra, but run tests and only upload if the tests
pass.
In addition, it should (if at all possible) be a windows bot, since the zip file
is pretty much only used by windows devs.

[iannucci, 2016-04-01 17:54:17]

On 2016/04/01 at 17:34:17, agable wrote:
> Big top-level comment:
> You're putting this on the chromium.infra waterfall, but it doesn't run any
tests. In my opinion, either:
> a) This should be on chromium.infra.cron, since it does a thing that should be
done regularly; or

It polls depot_tools, it's not a cron job

> b) This should be on chromium.infra, but run tests and only upload if the
tests pass.

The tests are meaningless and don't even work on windows.

> In addition, it should (if at all possible) be a windows bot, since the zip
file is pretty much only used by windows devs.

Building this zip on windows is much more painful than on linux, and would
reduce the relocatability of this recipe.

[iannucci, 2016-04-01 17:54:56]

On 2016/04/01 at 17:54:17, iannucci wrote:
> On 2016/04/01 at 17:34:17, agable wrote:
> > Big top-level comment:
> > You're putting this on the chromium.infra waterfall, but it doesn't run any
tests. In my opinion, either:
> > a) This should be on chromium.infra.cron, since it does a thing that should
be done regularly; or
> 
> It polls depot_tools, it's not a cron job
> 
> > b) This should be on chromium.infra, but run tests and only upload if the
tests pass.
> 
> The tests are meaningless and don't even work on windows.
> 
> > In addition, it should (if at all possible) be a windows bot, since the zip
file is pretty much only used by windows devs.
> 
> Building this zip on windows is much more painful than on linux, and would
reduce the relocatability of this recipe.

(and I've already spent a week and a half cleaning shit up for what should have
been a 15 minute migration. I have better things to do.)

[iannucci, 2016-04-01 18:01:36]

https://chromiumcodereview.appspot.com/1849113003/diff/20001/scripts/slave/re...
File scripts/slave/recipes/infra/depot_tools_builder.py (right):

https://chromiumcodereview.appspot.com/1849113003/diff/20001/scripts/slave/re...
scripts/slave/recipes/infra/depot_tools_builder.py:42: api.git('clone',
'--single-branch', '-n', REPO_URL, api.path['checkout'])
On 2016/04/01 at 14:40:01, dnj wrote:
> nit: Maybe nested step here?

done and above

https://chromiumcodereview.appspot.com/1849113003/diff/20001/scripts/slave/re...
scripts/slave/recipes/infra/depot_tools_builder.py:48: api.git('reflog',
'expire', '--all')
On 2016/04/01 at 14:40:01, dnj wrote:
> Might be worth adding this sort of thing to the Git recipe module, "compact"
or something, b/c I can see it being useful elsewhere.

I'm actually not sure where else it would be useful: we don't really archive
repos that often, and in the general case, autogc is fine. Did you have an
example in mind?

https://chromiumcodereview.appspot.com/1849113003/diff/20001/scripts/slave/re...
scripts/slave/recipes/infra/depot_tools_builder.py:59:
'gs://chrome-infra-docs/flat/depot_tools/docs/'],
On 2016/04/01 at 14:40:01, dnj wrote:
> nit: make this a constant up top

done

[iannucci, 2016-04-01 18:01:48]

The CQ bit was checked by iannucci@chromium.org

[commit-bot: I haz the power, 2016-04-01 18:01:55]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1849113003/20001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1849113003/20001

[commit-bot: I haz the power, 2016-04-01 18:05:55]

Description was changed from

==========
Add depot_tools builder to infra waterfall.

This builder clones, packs, and zips depot_tools for the purpose of
updating the windows depot_tools.zip bootstrap, and it also uploads the
generated `man/html` documentation folder to the chrome-infra-docs
bucket.

R=agable@chromium.org, dnj@chromium.org
BUG=598958
==========

to

==========
Add depot_tools builder to infra waterfall.

This builder clones, packs, and zips depot_tools for the purpose of
updating the windows depot_tools.zip bootstrap, and it also uploads the
generated `man/html` documentation folder to the chrome-infra-docs
bucket.

R=agable@chromium.org, dnj@chromium.org
BUG=598958

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299636
==========

[commit-bot: I haz the power, 2016-04-01 18:05:56]

Committed patchset #2 (id:20001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=299636

[agable, 2016-04-01 18:30:59]

Totally fine with this landing, just want to more clearly make my case:

On 2016/04/01 at 17:54:17, iannucci wrote:
> On 2016/04/01 at 17:34:17, agable wrote:
> > Big top-level comment:
> > You're putting this on the chromium.infra waterfall, but it doesn't run any
tests. In my opinion, either:
> > a) This should be on chromium.infra.cron, since it does a thing that should
be done regularly; or
> 
> It polls depot_tools, it's not a cron job

It doesn't have to poll depot_tools. It could run once an hour, or once a day,
and get essentially the same utility. And since this isn't compiling or testing
anything, it is pretty out of place on a CI waterfall.

> > b) This should be on chromium.infra, but run tests and only upload if the
tests pass.
> 
> The tests are meaningless and don't even work on windows.

So let's blacklist / whitelist the tests. Then we'll at least have a goal to
work towards.

> > In addition, it should (if at all possible) be a windows bot, since the zip
file is pretty much only used by windows devs.
> 
> Building this zip on windows is much more painful than on linux, and would
reduce the relocatability of this recipe.

Yeah, the "(if at all possible)" was because I was 99% sure that was a
infeasible long shot.

Anyway, totally understand what you're doing here, and *really* understand the
desire to just be done with it. It just seemed like this CL was a really good
time to get a head start on actually testing depot_tools like a real repo.