Script to generate buildbot json for both the perf waterfall and fyi

Script to generate buildbot json for both the perf waterfall and fyi
waterfall.

chromium.perf.json: This file should not change in overall content
although some of the ordering may be off.  I have also explicitly
set the shards for Win 7 Intel GPU Perf given the format my script
expects, which is the same as the current json file which doesn't
specify a shard.

Note: I have removed entries for bots that are not longer on our waterfall.  These include Win 7 Low-End Perf, Mac 10.9 and Mac 10.8.  This is simply cleanup and doesn't impact any functionality.

chromium.perf.fyi.json: This file is new and doesn't have any
configuration yet that will actually read this file.  The next
step is to trigger the jobs locally before I change the fyi recipes
to read from this json file.  Note: telemetry_perf_tests will be the
name of the gn target in a follow up cl.  Note that this file
will probably change as I iterate on getting the perf tests swarmed.

BUG=chromium:633253
Committed: https://crrev.com/0696f0e4b55d0d03a11baa66a798e1cd2d616fe5
Cr-Commit-Position: refs/heads/master@{#415133}

[eyaich1, 2016-08-24 18:14:13]

Description was changed from

==========
Script to generate buildbot json for both the perf waterfall and fyi
waterfall.

chromium.perf.json: This file should not change in overall content
although some of the ordering may be off.  I have also explicitly
set the shards for Win 7 Intel GPU Perf given the format my script
expects, which is the same as the current json file which doesn't
specify a shard.

chromium.perf.fyi.json: This file is new and doesn't have any
configuration yet that will actually read this file.  The next
step is to trigger the jobs locally before I change the fyi recipes
to read from this json file.  Note: telemetry_perf_tests will be the
name of the gn target in a follow up cl.  Note that this file
will probably change as I iterate on getting the perf tests swarmed.

BUG=chromium:633253
==========

to

==========
Script to generate buildbot json for both the perf waterfall and fyi
waterfall.

chromium.perf.json: This file should not change in overall content
although some of the ordering may be off.  I have also explicitly
set the shards for Win 7 Intel GPU Perf given the format my script
expects, which is the same as the current json file which doesn't
specify a shard.

Note: I have removed entries for bots that are not longer on our waterfall. 
These include Win 7 Low-End Perf, Mac 10.9 and Mac 10.8.  This is simply cleanup
and doesn't impact any functionality.

chromium.perf.fyi.json: This file is new and doesn't have any
configuration yet that will actually read this file.  The next
step is to trigger the jobs locally before I change the fyi recipes
to read from this json file.  Note: telemetry_perf_tests will be the
name of the gn target in a follow up cl.  Note that this file
will probably change as I iterate on getting the perf tests swarmed.

BUG=chromium:633253
==========

[eyaich1, 2016-08-24 18:25:20]

eyaich@chromium.org changed reviewers:
+ dtu@chromium.org

[eyaich1, 2016-08-24 18:25:20]

Like I mentioned in the notes this will be an iterative process to get the two
low end laptops running the perf tests.

Dave please do a full review.

Ken, if you could please do a high level sanity check on the script given that
you wrote a similar one on the gpu side I would greatly appreciate it.

[Ken Russell (switch to Gerrit), 2016-08-25 01:26:16]

kbr@chromium.org changed reviewers:
+ kbr@chromium.org

[Ken Russell (switch to Gerrit), 2016-08-25 01:26:17]

Excellent work. This will not only make these bots easier to maintain, but also
pave the way for auto-generating the rest of the JSON files in this directory.

I'd like to suggest first committing a CL which normalizes the contents of
chromium.perf.json -- removing old bots, alphabetizing entries -- so that this
CL essentially only adds the autogeneration header at the top.

However, LGTM overall. One question.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
File tools/perf/generate_perf_json.py (right):

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:104: 'shards': [2]
Out of curiosity, why is only shard 2 for this bot covered by
chromium.perf.json? Where do the other 4 shards of this bot pick up their test
descriptions? Same basic question for some of the others.

[dtu, 2016-08-25 01:30:23]

On 2016/08/25 01:26:17, Ken Russell wrote:
> Excellent work. This will not only make these bots easier to maintain, but
also
> pave the way for auto-generating the rest of the JSON files in this directory.
> 
> I'd like to suggest first committing a CL which normalizes the contents of
> chromium.perf.json -- removing old bots, alphabetizing entries -- so that this
> CL essentially only adds the autogeneration header at the top.
> 
> However, LGTM overall. One question.
> 
>
https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
> File tools/perf/generate_perf_json.py (right):
> 
>
https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
> tools/perf/generate_perf_json.py:104: 'shards': [2]
> Out of curiosity, why is only shard 2 for this bot covered by
> chromium.perf.json? Where do the other 4 shards of this bot pick up their test
> descriptions? Same basic question for some of the others.


Not too keen on having to update two repos every time a bot is added. Is this
the only/best way?

The original chromium.perf.json was definitely both incorrect and out of date. I
think what we really want in the end is something that expresses ideas like
"gpu_perftests runs on shard 1 of all Android bots" and "cc_perftests runs on
shard 1 of all bots." The non-Telemetry tests don't ever need to run on anything
other than shard 1.

Currently, the rest of the tests come from the config on the recipe side, which
calls "src/tools/perf/run_benchmark list" to generate the rest of the tests.
This means that whenever a new Telemetry benchmark is added, it automatically
gets run.

BTW I have a change in flight that will make FYI use chromium.perf.fyi.json.
https://codereview.chromium.org/2270563005/

[eyaich1, 2016-08-25 13:09:38]

On 2016/08/25 01:30:23, dtu wrote:
> On 2016/08/25 01:26:17, Ken Russell wrote:
> > Excellent work. This will not only make these bots easier to maintain, but
> also
> > pave the way for auto-generating the rest of the JSON files in this
directory.
> > 
> > I'd like to suggest first committing a CL which normalizes the contents of
> > chromium.perf.json -- removing old bots, alphabetizing entries -- so that
this
> > CL essentially only adds the autogeneration header at the top.

https://codereview.chromium.org/2277003004/ is out for review that should take
of this.


> > 
> > However, LGTM overall. One question.
> > 
> >
>
https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
> > File tools/perf/generate_perf_json.py (right):
> > 
> >
>
https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
> > tools/perf/generate_perf_json.py:104: 'shards': [2]
> > Out of curiosity, why is only shard 2 for this bot covered by
> > chromium.perf.json? Where do the other 4 shards of this bot pick up their
test
> > descriptions? Same basic question for some of the others.
> 
> 
> Not too keen on having to update two repos every time a bot is added. Is this
> the only/best way?

Yes as far as I know there is no way to read from a common config file for this.
 This is how
they do it for the GPU script.  He indicated a while back that this wasn't a
huge issue for them
but I am not sure how often these get updated in our configs.  For now we just
need to have
comments in both places to make sure people who are updating them are aware.

> 
> The original chromium.perf.json was definitely both incorrect and out of date.
I
> think what we really want in the end is something that expresses ideas like
> "gpu_perftests runs on shard 1 of all Android bots" and "cc_perftests runs on
> shard 1 of all bots." The non-Telemetry tests don't ever need to run on
anything
> other than shard 1.

As I noted in Ken's comment, I think this should be saved for a follow on CL. 
This CL 
was just to auto generate what is already there.


> 
> Currently, the rest of the tests come from the config on the recipe side,
which
> calls "src/tools/perf/run_benchmark list" to generate the rest of the tests.
> This means that whenever a new Telemetry benchmark is added, it automatically
> gets run.

I will be changing this.  If you look at my design doc for swarming the perf
waterfall
(https://docs.google.com/document/d/12L1AXbkgL9wJMe_a2YfLCBvZwQKmwBJdEUPVaFT7i...)
we have decided to go to static generation of this so all benchmarks will run on
all 
bots.  We are then changing the telemetry logic when running disabled tests 
( see https://codereview.chromium.org/2265423005/).

> 
> BTW I have a change in flight that will make FYI use chromium.perf.fyi.json.
> https://codereview.chromium.org/2270563005/

This change doesn't appear to contain the FYI json file.  It simply changes
the way the recipe handles the DynamicPerfTests call unless I am missing
something.

[eyaich1, 2016-08-25 13:09:46]

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
File tools/perf/generate_perf_json.py (right):

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:104: 'shards': [2]
On 2016/08/25 01:26:17, Ken Russell wrote:
> Out of curiosity, why is only shard 2 for this bot covered by
> chromium.perf.json? Where do the other 4 shards of this bot pick up their test
> descriptions? Same basic question for some of the others.

Like Dave noted I think this is somewhat random and should be standardized.  For
this CL I am just trying to generate what is there, I think the standardization
should be done in a follow on CL.

[dtu, 2016-08-25 17:39:05]

Looks good overall. Mostly style issues.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
File tools/perf/generate_perf_json.py (right):

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:20: 
style guide: two spaces between all top level blocks

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:199: num_host_shards=1, num_device_shards=1,
swarming=None):
style guide: align indent with parameters on the first line

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:202: 'platform': platform,
style guide: 4-space indent

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:207: if swarming is not None:
style guide: implicit false. if swarming:

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:216: def generateFyiWaterfall():
style guide: lower_with_under() or CapWords()

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:219: 'win-low-end-2-core', 'win',
style guide: align indent with parameters on first line

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:290: ]
We need --device=android on android.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:318: for enabled_shard in
enabled_tester['shards']:
if shard in enabled_tester['shards']:
  return True

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:341: elif tester_config['platform'] == 'win' \
style guide: don't use \. wrap in parentheses.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:385: if len(scripts) > 0:
style guide: implicit false. if scripts:

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:402: waterfall = generateWaterfall()
It's inconsistent that generateWaterfall and generate_all_tests have similar
names, but one returns its result and the other writes its result to a file.

[dtu, 2016-08-25 17:40:50]

On 2016/08/25 13:09:38, eyaich1 wrote:
> On 2016/08/25 01:30:23, dtu wrote:
> > On 2016/08/25 01:26:17, Ken Russell wrote:
> > > Excellent work. This will not only make these bots easier to maintain, but
> > also
> > > pave the way for auto-generating the rest of the JSON files in this
> directory.
> > > 
> > > I'd like to suggest first committing a CL which normalizes the contents of
> > > chromium.perf.json -- removing old bots, alphabetizing entries -- so that
> this
> > > CL essentially only adds the autogeneration header at the top.
> 
> https://codereview.chromium.org/2277003004/ is out for review that should take
> of this.
> 
> 
> > > 
> > > However, LGTM overall. One question.
> > > 
> > >
> >
>
https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
> > > File tools/perf/generate_perf_json.py (right):
> > > 
> > >
> >
>
https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
> > > tools/perf/generate_perf_json.py:104: 'shards': [2]
> > > Out of curiosity, why is only shard 2 for this bot covered by
> > > chromium.perf.json? Where do the other 4 shards of this bot pick up their
> test
> > > descriptions? Same basic question for some of the others.
> > 
> > 
> > Not too keen on having to update two repos every time a bot is added. Is
this
> > the only/best way?
> 
> Yes as far as I know there is no way to read from a common config file for
this.
>  This is how
> they do it for the GPU script.  He indicated a while back that this wasn't a
> huge issue for them
> but I am not sure how often these get updated in our configs.  For now we just
> need to have
> comments in both places to make sure people who are updating them are aware.

Okay.

> > The original chromium.perf.json was definitely both incorrect and out of
date.
> I
> > think what we really want in the end is something that expresses ideas like
> > "gpu_perftests runs on shard 1 of all Android bots" and "cc_perftests runs
on
> > shard 1 of all bots." The non-Telemetry tests don't ever need to run on
> anything
> > other than shard 1.
> 
> As I noted in Ken's comment, I think this should be saved for a follow on CL. 
> This CL 
> was just to auto generate what is already there.

Sounds good.

> > Currently, the rest of the tests come from the config on the recipe side,
> which
> > calls "src/tools/perf/run_benchmark list" to generate the rest of the tests.
> > This means that whenever a new Telemetry benchmark is added, it
automatically
> > gets run.
> 
> I will be changing this.  If you look at my design doc for swarming the perf
> waterfall
>
(https://docs.google.com/document/d/12L1AXbkgL9wJMe_a2YfLCBvZwQKmwBJdEUPVaFT7i...)
> we have decided to go to static generation of this so all benchmarks will run
on
> all 
> bots.  We are then changing the telemetry logic when running disabled tests 
> ( see https://codereview.chromium.org/2265423005/).

Cool. I think this gives us more leeway to play around with the way sharding is
done too.

> > BTW I have a change in flight that will make FYI use chromium.perf.fyi.json.
> > https://codereview.chromium.org/2270563005/
> 
> This change doesn't appear to contain the FYI json file.  It simply changes
> the way the recipe handles the DynamicPerfTests call unless I am missing
> something.

It implicitly does by deleting the "test_spec_file" field, which causes it to
revert to its default value. But there's no isolate test_generator set, so I
think it's still a noop.

[eyaich1, 2016-08-29 18:38:10]

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
File tools/perf/generate_perf_json.py (right):

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:20: 
On 2016/08/25 17:39:05, dtu wrote:
> style guide: two spaces between all top level blocks

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:199: num_host_shards=1, num_device_shards=1,
swarming=None):
On 2016/08/25 17:39:05, dtu wrote:
> style guide: align indent with parameters on the first line

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:202: 'platform': platform,
On 2016/08/25 17:39:05, dtu wrote:
> style guide: 4-space indent

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:207: if swarming is not None:
On 2016/08/25 17:39:05, dtu wrote:
> style guide: implicit false. if swarming:

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:216: def generateFyiWaterfall():
On 2016/08/25 17:39:05, dtu wrote:
> style guide: lower_with_under() or CapWords()

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:219: 'win-low-end-2-core', 'win',
On 2016/08/25 17:39:05, dtu wrote:
> style guide: align indent with parameters on first line

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:290: ]
On 2016/08/25 17:39:05, dtu wrote:
> We need --device=android on android.

We don't need it yet, I added a comment.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:318: for enabled_shard in
enabled_tester['shards']:
On 2016/08/25 17:39:05, dtu wrote:
> if shard in enabled_tester['shards']:
>   return True

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:341: elif tester_config['platform'] == 'win' \
On 2016/08/25 17:39:05, dtu wrote:
> style guide: don't use \. wrap in parentheses.

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:385: if len(scripts) > 0:
On 2016/08/25 17:39:05, dtu wrote:
> style guide: implicit false. if scripts:

Done.

https://codereview.chromium.org/2272603005/diff/40001/tools/perf/generate_per...
tools/perf/generate_perf_json.py:402: waterfall = generateWaterfall()
On 2016/08/25 17:39:05, dtu wrote:
> It's inconsistent that generateWaterfall and generate_all_tests have similar
> names, but one returns its result and the other writes its result to a file.

Done.

[dtu, 2016-08-29 18:43:51]

lgtm

[eyaich1, 2016-08-29 22:42:16]

The CQ bit was checked by eyaich@chromium.org

[eyaich1, 2016-08-29 22:42:17]

The patchset sent to the CQ was uploaded after l-g-t-m from kbr@chromium.org
Link to the patchset: https://codereview.chromium.org/2272603005/#ps80001
(title: "Removing all win 7 intel gpu perf shards")

[commit-bot: I haz the power, 2016-08-29 23:21:08]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-30 05:05:14]

Description was changed from

==========
Script to generate buildbot json for both the perf waterfall and fyi
waterfall.

chromium.perf.json: This file should not change in overall content
although some of the ordering may be off.  I have also explicitly
set the shards for Win 7 Intel GPU Perf given the format my script
expects, which is the same as the current json file which doesn't
specify a shard.

Note: I have removed entries for bots that are not longer on our waterfall. 
These include Win 7 Low-End Perf, Mac 10.9 and Mac 10.8.  This is simply cleanup
and doesn't impact any functionality.

chromium.perf.fyi.json: This file is new and doesn't have any
configuration yet that will actually read this file.  The next
step is to trigger the jobs locally before I change the fyi recipes
to read from this json file.  Note: telemetry_perf_tests will be the
name of the gn target in a follow up cl.  Note that this file
will probably change as I iterate on getting the perf tests swarmed.

BUG=chromium:633253
==========

to

==========
Script to generate buildbot json for both the perf waterfall and fyi
waterfall.

chromium.perf.json: This file should not change in overall content
although some of the ordering may be off.  I have also explicitly
set the shards for Win 7 Intel GPU Perf given the format my script
expects, which is the same as the current json file which doesn't
specify a shard.

Note: I have removed entries for bots that are not longer on our waterfall. 
These include Win 7 Low-End Perf, Mac 10.9 and Mac 10.8.  This is simply cleanup
and doesn't impact any functionality.

chromium.perf.fyi.json: This file is new and doesn't have any
configuration yet that will actually read this file.  The next
step is to trigger the jobs locally before I change the fyi recipes
to read from this json file.  Note: telemetry_perf_tests will be the
name of the gn target in a follow up cl.  Note that this file
will probably change as I iterate on getting the perf tests swarmed.

BUG=chromium:633253
==========

[commit-bot: I haz the power, 2016-08-30 05:05:15]

Committed patchset #5 (id:80001)

[commit-bot: I haz the power, 2016-08-30 05:07:34]

Description was changed from

==========
Script to generate buildbot json for both the perf waterfall and fyi
waterfall.

chromium.perf.json: This file should not change in overall content
although some of the ordering may be off.  I have also explicitly
set the shards for Win 7 Intel GPU Perf given the format my script
expects, which is the same as the current json file which doesn't
specify a shard.

Note: I have removed entries for bots that are not longer on our waterfall. 
These include Win 7 Low-End Perf, Mac 10.9 and Mac 10.8.  This is simply cleanup
and doesn't impact any functionality.

chromium.perf.fyi.json: This file is new and doesn't have any
configuration yet that will actually read this file.  The next
step is to trigger the jobs locally before I change the fyi recipes
to read from this json file.  Note: telemetry_perf_tests will be the
name of the gn target in a follow up cl.  Note that this file
will probably change as I iterate on getting the perf tests swarmed.

BUG=chromium:633253
==========

to

==========
Script to generate buildbot json for both the perf waterfall and fyi
waterfall.

chromium.perf.json: This file should not change in overall content
although some of the ordering may be off.  I have also explicitly
set the shards for Win 7 Intel GPU Perf given the format my script
expects, which is the same as the current json file which doesn't
specify a shard.

Note: I have removed entries for bots that are not longer on our waterfall. 
These include Win 7 Low-End Perf, Mac 10.9 and Mac 10.8.  This is simply cleanup
and doesn't impact any functionality.

chromium.perf.fyi.json: This file is new and doesn't have any
configuration yet that will actually read this file.  The next
step is to trigger the jobs locally before I change the fyi recipes
to read from this json file.  Note: telemetry_perf_tests will be the
name of the gn target in a follow up cl.  Note that this file
will probably change as I iterate on getting the perf tests swarmed.

BUG=chromium:633253

Committed: https://crrev.com/0696f0e4b55d0d03a11baa66a798e1cd2d616fe5
Cr-Commit-Position: refs/heads/master@{#415133}
==========

[commit-bot: I haz the power, 2016-08-30 05:07:36]

Patchset 5 (id:??) landed as
https://crrev.com/0696f0e4b55d0d03a11baa66a798e1cd2d616fe5
Cr-Commit-Position: refs/heads/master@{#415133}