[Telemetry] Add Benchmark.ShouldTearDownStateAfterEachStorySetRun

[Telemetry] Add Benchmark.ShouldTearDownStateAfterEachStorySetRun

Defaults to True in order to reset the state and make individual story set
repeats more independent of each other. The intended effect is to average
out noise in measurements between repeats.

Long running benchmarks willing to stess test the browser and have it run
for long periods of time may switch this value to False.

BUG=catapult:#2294

Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/310495788cee041306bcfc0eb289c0184d47aca2

[perezju, 2016-06-24 10:09:13]

perezju@chromium.org changed reviewers:
+ eakuefner@chromium.org, nednguyen@google.com

[perezju, 2016-06-24 10:12:32]

https://codereview.chromium.org/2091103003/diff/1/telemetry/telemetry/benchma...
File telemetry/telemetry/benchmark_unittest.py (right):

https://codereview.chromium.org/2091103003/diff/1/telemetry/telemetry/benchma...
telemetry/telemetry/benchmark_unittest.py:141: *args, **kwargs): # pylint:
disable=unused-argument
fixing some random issue I found while making sure that my change wouldn't break
other clients' expectations.

[nednguyen, 2016-06-24 14:44:51]

https://codereview.chromium.org/2091103003/diff/1/telemetry/telemetry/benchma...
File telemetry/telemetry/benchmark.py (right):

https://codereview.chromium.org/2091103003/diff/1/telemetry/telemetry/benchma...
telemetry/telemetry/benchmark.py:129: return True
This will cause a behavior change to all existing benchmarks that's using
pageset-repeat. I think we should make this False for now, then send an email to
telemetry-announce@ telling people about this, then switch this to True.

[perezju, 2016-06-24 15:12:36]

https://codereview.chromium.org/2091103003/diff/1/telemetry/telemetry/benchma...
File telemetry/telemetry/benchmark.py (right):

https://codereview.chromium.org/2091103003/diff/1/telemetry/telemetry/benchma...
telemetry/telemetry/benchmark.py:129: return True
On 2016/06/24 14:44:50, nednguyen wrote:
> This will cause a behavior change to all existing benchmarks that's using
> pageset-repeat. I think we should make this False for now, then send an email
to
> telemetry-announce@ telling people about this, then switch this to True.

ok, sgtm

[nednguyen, 2016-06-24 16:11:55]

lgtm

https://codereview.chromium.org/2091103003/diff/20001/telemetry/telemetry/ben...
File telemetry/telemetry/benchmark.py (right):

https://codereview.chromium.org/2091103003/diff/20001/telemetry/telemetry/ben...
telemetry/telemetry/benchmark.py:131: #
https://github.com/catapult-project/catapult/issues/2294#issuecomment-228306977
this may cause a lint check failure. Feel free to add s.t similar to
https://cs.chromium.org/chromium/src/tools/perf/pylintrc?rcl=0&l=27

[perezju, 2016-06-27 08:11:10]

The CQ bit was checked by perezju@chromium.org

[commit-bot: I haz the power, 2016-06-27 08:11:15]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-06-27 08:33:52]

Description was changed from

==========
[Telemetry] Add Benchmark.ShouldTearDownStateAfterEachStorySetRun

Defaults to True in order to reset the state and make individual story set
repeats more independent of each other. The intended effect is to average
out noise in measurements between repeats.

Long running benchmarks willing to stess test the browser and have it run
for long periods of time may switch this value to False.

BUG=catapult:#2294
==========

to

==========
[Telemetry] Add Benchmark.ShouldTearDownStateAfterEachStorySetRun

Defaults to True in order to reset the state and make individual story set
repeats more independent of each other. The intended effect is to average
out noise in measurements between repeats.

Long running benchmarks willing to stess test the browser and have it run
for long periods of time may switch this value to False.

BUG=catapult:#2294

Committed:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...
==========

[commit-bot: I haz the power, 2016-06-27 08:33:53]

Committed patchset #2 (id:20001) as
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...