Run non-rendering benches in their own config.

Run non-rendering benches in their own config.

Currently benches that set fIsRendering = false are run as part of the first config run. This is problematic for several reasons.

1) Which config they are run as depends on the --config options passed to bench. If I run bench --config GPU --config 8888 then they will be run in the GPU config and not the 8888 config.

2) Their presence makes bench take longer to run when testing rendering changes or comparing one rendering config to another (e.g. GPU v 8888). This is especially true on Android.

3) When comparing runs of a single config across multiple bench runs (typically  with code changes) it isn't obvious from the output which benchs to ignore because they say are listed under as being part of the config.

This CL adds NONRENDERING as a config (open to name suggestions). Non-rendering tests are run in and only in the NONRENDERING config.

Will this screw up our bench bots/scripts/graphs? If so I think we should change them since running non-rendering tests separately makes a lot more sense than the current madness.
Committed: https://code.google.com/p/skia/source/detail?r=8174

[benchen, 2013-03-14 20:23:04]

Sounds to me that the changes to extra flags and multiple configs are mostly for
developers' manual explorations. If so, and as long as bots don't change their
ways of using the flags, the bench runs and outputs will stay the same, which
won't affect jobs that depend on the current formats of bench log outputs.

[bsalomon, 2013-03-14 20:27:43]

On 2013/03/14 20:23:04, benchen wrote:
> Sounds to me that the changes to extra flags and multiple configs are mostly
for
> developers' manual explorations. If so, and as long as bots don't change their
> ways of using the flags, the bench runs and outputs will stay the same, which
> won't affect jobs that depend on the current formats of bench log outputs.

The format of the output will remain the same but there will be an extra config
and many of the existing benchs will only show up in this new config (as opposed
to whatever config they were showing up in before, probably 8888).

[benchen, 2013-03-14 20:37:50]

Oh, so the config names will be different. If it's for microbenches and won't
affect picture benches, we're at least fine with the skp side. For microbenches,
it'll mean newly added config names. Maybe I can clean up old ones after the
transition is done.
BTW just found that the bigtable hosting microbenche data are down for PCR
maintenance. Glad that at least I switched skp benches to use MoB backend.
Microbenches should be up again tomorrow.
On 2013/03/14 20:27:43, bsalomon wrote:
> On 2013/03/14 20:23:04, benchen wrote:
> > Sounds to me that the changes to extra flags and multiple configs are mostly
> for
> > developers' manual explorations. If so, and as long as bots don't change
their
> > ways of using the flags, the bench runs and outputs will stay the same,
which
> > won't affect jobs that depend on the current formats of bench log outputs.
> 
> The format of the output will remain the same but there will be an extra
config
> and many of the existing benchs will only show up in this new config (as
opposed
> to whatever config they were showing up in before, probably 8888).

[robertphillips, 2013-03-14 21:13:51]

lgtm

https://codereview.chromium.org/12475012/diff/1/bench/benchmain.cpp
File bench/benchmain.cpp (right):

https://codereview.chromium.org/12475012/diff/1/bench/benchmain.cpp#newcode331
bench/benchmain.cpp:331: static const int kConfigCount =
SK_ARRAY_COUNT(gConfigs);
size_t -> int? or kConfigCount -> size_t?

[benchen, 2013-03-14 21:31:34]

LGTM. Just double-checked bench_util.py: please make sure the new config names
do not include empty space, and do not end with "]" or ">".

On 2013/03/14 21:13:51, robertphillips wrote:
> lgtm
> 
> https://codereview.chromium.org/12475012/diff/1/bench/benchmain.cpp
> File bench/benchmain.cpp (right):
> 
> https://codereview.chromium.org/12475012/diff/1/bench/benchmain.cpp#newcode331
> bench/benchmain.cpp:331: static const int kConfigCount =
> SK_ARRAY_COUNT(gConfigs);
> size_t -> int? or kConfigCount -> size_t?

[borenet, 2013-03-14 21:33:25]

Should we be running bench on the bots with any --config?  Right now we aren't
using any.

[bsalomon, 2013-03-15 15:42:24]

Committed patchset #1 manually as r8174 (presubmit successful).

[bsalomon, 2013-03-15 15:44:10]

https://codereview.chromium.org/12475012/diff/1/bench/benchmain.cpp
File bench/benchmain.cpp (right):

https://codereview.chromium.org/12475012/diff/1/bench/benchmain.cpp#newcode331
bench/benchmain.cpp:331: static const int kConfigCount =
SK_ARRAY_COUNT(gConfigs);
On 2013/03/14 21:13:51, robertphillips wrote:
> size_t -> int? or kConfigCount -> size_t?

Done. (kConfigCount -> size_t).