add recipe continuous builders for each repo with recipes

add recipe continuous builders for each repo with recipes

Adds continuous builders for each repository which currently contains recipes. See bug for more details.

depends on https://codereview.chromium.org/2061263003

BUG=614200
Committed: https://chromium.googlesource.com/chromium/tools/build/+/81f9c862dade711affea7a586b6a2bffdf219cc9

[martiniss, 2016-06-11 02:20:29]

martiniss@chromium.org changed reviewers:
+ vadimsh@chromium.org

[martiniss, 2016-06-11 02:20:30]

Question for you

not ready for review

https://codereview.chromium.org/2064453002/diff/1/scripts/slave/recipes/infra...
File scripts/slave/recipes/infra/recipe_simulation.py (right):

https://codereview.chromium.org/2064453002/diff/1/scripts/slave/recipes/infra...
scripts/slave/recipes/infra/recipe_simulation.py:17: def _python(self, name,
script, args, **kwargs):
ignore this

https://codereview.chromium.org/2064453002/diff/1/scripts/slave/recipes/infra...
scripts/slave/recipes/infra/recipe_simulation.py:56: recipes_cfg =
parse_protobuf(result['content'].split('\n'))
should i move this function to the luci_config recipe module? is anything other
than a pseudo proto ever stored in luci-config?

Implementation is
https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/s...

[Vadim Sh., 2016-06-11 02:25:05]

https://codereview.chromium.org/2064453002/diff/1/scripts/slave/recipes/infra...
File scripts/slave/recipes/infra/recipe_simulation.py (right):

https://codereview.chromium.org/2064453002/diff/1/scripts/slave/recipes/infra...
scripts/slave/recipes/infra/recipe_simulation.py:56: recipes_cfg =
parse_protobuf(result['content'].split('\n'))
On 2016/06/11 02:20:30, martiniss wrote:
> should i move this function to the luci_config recipe module? is anything
other
> than a pseudo proto ever stored in luci-config?
> 
> Implementation is
>
https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/s...

luci-config can store any kinds of smallish files, not necessary protos.

[martiniss, 2016-06-11 02:33:14]

https://codereview.chromium.org/2064453002/diff/1/scripts/slave/recipes/infra...
File scripts/slave/recipes/infra/recipe_simulation.py (right):

https://codereview.chromium.org/2064453002/diff/1/scripts/slave/recipes/infra...
scripts/slave/recipes/infra/recipe_simulation.py:56: recipes_cfg =
parse_protobuf(result['content'].split('\n'))
On 2016/06/11 at 02:25:05, Vadim Sh. wrote:
> On 2016/06/11 02:20:30, martiniss wrote:
> > should i move this function to the luci_config recipe module? is anything
other
> > than a pseudo proto ever stored in luci-config?
> > 
> > Implementation is
> >
https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/s...
> 
> luci-config can store any kinds of smallish files, not necessary protos.

Ok. WDYT of adding something like luci_config.get_project_proto_config, which
does this fake proto parsing? 

It basically turns the proto into a python dict. It should do some parsing given
the proto file, but i'm not sure how I would do that.... maybe build protos into
the recipe engine.

[martiniss, 2016-06-14 01:22:40]

martiniss@chromium.org changed reviewers:
+ iannucci@chromium.org, tandrii@chromium.org

[martiniss, 2016-06-14 01:22:40]

PTAL

vadimsh for luci_config recipe module interface
iannucci for general
tandrii for recipe_simulation recipe

cc estaab FYI

[tandrii(chromium), 2016-06-14 12:22:51]

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/luci_config/example.py (right):

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/luci_config/example.py:44: ]
why not """
  foo: 1
...
""" it's easier to read

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
File scripts/slave/recipes/infra/recipe_simulation.py (right):

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/recipe_simulation.py:4: """A continious builder for
build repo which simulates recipes."""
update doc

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/recipe_simulation.py:28: # Needed to set up the
infra checkout for infra python, which has coverage.
bleh, isn't that slow every time running runhooks to install 100s of packages
just to get 1? At least add a TODO, but ideally how about this:
* copy-paste bootstrap dir into recipe engine (or new repo which just contains
the bootstrap and perhaps recipes.py)
* edit bootstrap/deps.pyl and leave only coverage

WDYT? This could actually be a solution to http://crbug.com/618709

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/recipe_simulation.py:30: c.solutions[0].revision =
'origin/master'
and why not a fixed revision? I don't see a need to keep moving *with* infra if
all you want is coverage.

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/recipe_simulation.py:34: cache_dir =
root_dir.join('_cache_dir')
remove line, it's a dup

[iannucci, 2016-06-14 21:41:43]

Please update the CL description to have more context. A bug link would be
really helpful at the very least.

[iannucci, 2016-06-14 21:46:10]

seems ok pending other comments

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/luci_config/api.py (right):

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/luci_config/api.py:77: def parse_protobuf(self,
lines):
can we rename this to `parse_textproto`? protobuf implies an actual protobuf
(i.e. binary).

[martiniss, 2016-06-14 21:57:55]

Description was changed from

==========
add recipe continuous builders for each repo with recipes
==========

to

==========
add recipe continuous builders for each repo with recipes

Adds continuous builders for each repository which currently contains recipes.
See bug for more details.

BUG=614200
==========

[martiniss, 2016-06-14 23:11:19]

Description was changed from

==========
add recipe continuous builders for each repo with recipes

Adds continuous builders for each repository which currently contains recipes.
See bug for more details.

BUG=614200
==========

to

==========
add recipe continuous builders for each repo with recipes

Adds continuous builders for each repository which currently contains recipes.
See bug for more details.

depends on https://codereview.chromium.org/2061263003

BUG=614200
==========

[martiniss, 2016-06-14 23:11:34]

ok, PTAL

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/luci_config/api.py (right):

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/luci_config/api.py:77: def parse_protobuf(self,
lines):
On 2016/06/14 at 21:46:09, iannucci wrote:
> can we rename this to `parse_textproto`? protobuf implies an actual protobuf
(i.e. binary).

Done.

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/luci_config/example.py (right):

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/luci_config/example.py:44: ]
On 2016/06/14 at 12:22:51, tandrii(chromium) wrote:
> why not """
>   foo: 1
> ...
> """ it's easier to read

Fancy! Done.

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
File scripts/slave/recipes/infra/recipe_simulation.py (right):

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/recipe_simulation.py:4: """A continious builder for
build repo which simulates recipes."""
On 2016/06/14 at 12:22:51, tandrii(chromium) wrote:
> update doc

Done.

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/recipe_simulation.py:28: # Needed to set up the
infra checkout for infra python, which has coverage.
On 2016/06/14 at 12:22:51, tandrii(chromium) wrote:
> bleh, isn't that slow every time running runhooks to install 100s of packages
just to get 1? At least add a TODO, but ideally how about this:
> * copy-paste bootstrap dir into recipe engine (or new repo which just contains
the bootstrap and perhaps recipes.py)
> * edit bootstrap/deps.pyl and leave only coverage
> 
> WDYT? This could actually be a solution to http://crbug.com/618709

Done in https://codereview.chromium.org/2061263003

https://codereview.chromium.org/2064453002/diff/80001/scripts/slave/recipes/i...
scripts/slave/recipes/infra/recipe_simulation.py:34: cache_dir =
root_dir.join('_cache_dir')
On 2016/06/14 at 12:22:51, tandrii(chromium) wrote:
> remove line, it's a dup

Done.

[tandrii(chromium), 2016-06-15 10:01:26]

https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
File masters/master.chromium.infra/master.cfg (right):

https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
masters/master.chromium.infra/master.cfg:48: 'origin/master')
what's this for?

[martiniss, 2016-06-15 21:48:32]

https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
File masters/master.chromium.infra/master.cfg (right):

https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
masters/master.chromium.infra/master.cfg:48: 'origin/master')
On 2016/06/15 at 10:01:26, tandrii(chromium) wrote:
> what's this for?

No clue? It's not mine?

[tandrii(chromium), 2016-06-16 15:32:33]

On 2016/06/15 21:48:32, martiniss wrote:
>
https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
> File masters/master.chromium.infra/master.cfg (right):
> 
>
https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
> masters/master.chromium.infra/master.cfg:48: 'origin/master')
> On 2016/06/15 at 10:01:26, tandrii(chromium) wrote:
> > what's this for?
> 
> No clue? It's not mine?

OOps, sorry, looked a tdiff between patches, which probably caught rebase
artifact.

[tandrii(chromium), 2016-06-16 15:39:36]

LGTM % you need to fix the master config.

https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
File masters/master.chromium.infra/master.cfg (right):

https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
masters/master.chromium.infra/master.cfg:281: 'recipes-py-continuous',
you want to add your builders to be scheduled here and elsewhere

[martiniss, 2016-06-16 21:26:31]

On 2016/06/16 at 15:39:36, tandrii wrote:
> LGTM % you need to fix the master config.
> 
>
https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
> File masters/master.chromium.infra/master.cfg (right):
> 
>
https://codereview.chromium.org/2064453002/diff/100001/masters/master.chromiu...
> masters/master.chromium.infra/master.cfg:281: 'recipes-py-continuous',
> you want to add your builders to be scheduled here and elsewhere

Done.

[martiniss, 2016-06-16 21:26:34]

The CQ bit was checked by martiniss@chromium.org

[martiniss, 2016-06-16 21:26:35]

The patchset sent to the CQ was uploaded after l-g-t-m from tandrii@chromium.org
Link to the patchset: https://codereview.chromium.org/2064453002/#ps140001
(title: "Fix presubmit")

[commit-bot: I haz the power, 2016-06-16 21:26:40]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/2064453002/140001

[commit-bot: I haz the power, 2016-06-16 21:30:43]

Description was changed from

==========
add recipe continuous builders for each repo with recipes

Adds continuous builders for each repository which currently contains recipes.
See bug for more details.

depends on https://codereview.chromium.org/2061263003

BUG=614200
==========

to

==========
add recipe continuous builders for each repo with recipes

Adds continuous builders for each repository which currently contains recipes.
See bug for more details.

depends on https://codereview.chromium.org/2061263003

BUG=614200

Committed:
https://chromium.googlesource.com/chromium/tools/build/+/81f9c862dade711affea...
==========

[commit-bot: I haz the power, 2016-06-16 21:30:44]

Committed patchset #8 (id:140001) as
https://chromium.googlesource.com/chromium/tools/build/+/81f9c862dade711affea...

[dsansome-google, 2017-03-03 03:48:23]

dsansome@google.com changed reviewers:
+ dsansome@google.com

[dsansome-google, 2017-03-03 03:48:24]

https://codereview.chromium.org/2064453002/diff/140001/scripts/slave/recipe_m...
File scripts/slave/recipe_modules/luci_config/api.py (right):

https://codereview.chromium.org/2064453002/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/luci_config/api.py:80: This is not real protobuf
parsing at the moment; eventually, maybe it could
ಠ_ಠ

[iannucci1, 2017-03-03 05:15:29]

iannucci@google.com changed reviewers:
+ iannucci@google.com

[iannucci1, 2017-03-03 05:15:30]

https://codereview.chromium.org/2064453002/diff/140001/scripts/slave/recipe_m...
File scripts/slave/recipe_modules/luci_config/api.py (right):

https://codereview.chromium.org/2064453002/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/luci_config/api.py:80: This is not real protobuf
parsing at the moment; eventually, maybe it could
On 2017/03/03 03:48:24, dsansome-google wrote:
> ಠ_ಠ

Yep, getting a protobuf library into all the environments where this needs to
run is actually pretty hard (without something like vpython). Did you run into a
bug with this?

[martiniss, 2017-03-03 06:05:55]

On 2017/03/03 at 05:15:30, iannucci wrote:
>
https://codereview.chromium.org/2064453002/diff/140001/scripts/slave/recipe_m...
> File scripts/slave/recipe_modules/luci_config/api.py (right):
> 
>
https://codereview.chromium.org/2064453002/diff/140001/scripts/slave/recipe_m...
> scripts/slave/recipe_modules/luci_config/api.py:80: This is not real protobuf
parsing at the moment; eventually, maybe it could
> On 2017/03/03 03:48:24, dsansome-google wrote:
> > ಠ_ಠ
> 
> Yep, getting a protobuf library into all the environments where this needs to
run is actually pretty hard (without something like vpython). Did you run into a
bug with this?

I had plans to move this to luci config. I have half a CL out there, but I
haven't gotten around to landing it.