[Telemetry] Switch ShouldTearDownStateAfterEachStoryRun to True as default

[Telemetry] Switch ShouldTearDownStateAfterEachStoryRun to True as default

To reduce complexity of benchmarks and imcrease stability of measurements
benchmarks will now tear down the state (i.e. restart the browser) between
each page/story of a test.

Perf sherfiffs warning: This CL is expected to affect measurements of
many benchmarks.

BUG=chromium:640990
Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/4f812c9035e6cf9bad6774bb6ff71c1822167e0a

[perezju, 2016-09-02 09:55:09]

perezju@chromium.org changed reviewers:
+ nednguyen@google.com

[perezju, 2016-09-02 10:46:58]

The CQ bit was checked by perezju@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-02 10:47:05]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-02 11:08:47]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-02 11:08:48]

Dry run: This issue passed the CQ dry run.

[nednguyen, 2016-09-02 12:46:21]

lgtm

Can you land on s.t like Saturday morning? :P

[perezju, 2016-09-02 12:57:27]

On 2016/09/02 12:46:21, nednguyen wrote:
> lgtm
> 
> Can you land on s.t like Saturday morning? :P

Yeah, I can do that :)

[perezju, 2016-09-03 10:43:42]

The CQ bit was checked by perezju@chromium.org

[commit-bot: I haz the power, 2016-09-03 10:43:52]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-03 10:45:29]

Description was changed from

==========
[Telemetry] Switch ShouldTearDownStateAfterEachStoryRun to True as default

To reduce complexity of benchmarks and imcrease stability of measurements
benchmarks will now tear down the state (i.e. restart the browser) between
each page/story of a test.

Perf sherfiffs warning: This CL is expected to affect measurements of
many benchmarks.

BUG=chromium:640990
==========

to

==========
[Telemetry] Switch ShouldTearDownStateAfterEachStoryRun to True as default

To reduce complexity of benchmarks and imcrease stability of measurements
benchmarks will now tear down the state (i.e. restart the browser) between
each page/story of a test.

Perf sherfiffs warning: This CL is expected to affect measurements of
many benchmarks.

BUG=chromium:640990

Committed:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...
==========

[commit-bot: I haz the power, 2016-09-03 10:45:30]

Committed patchset #1 (id:1) as
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

[Ting-Yuan (Leo) Huang, 2016-09-09 23:21:37]

This has made page_cycler much longer and telemetry_AFDOGenerate has been
failing for more than a week. Can we revert this for now?

[nednguyen, 2016-09-10 02:25:08]

On 2016/09/09 23:21:37, Ting-Yuan (Leo) Huang wrote:
> This has made page_cycler much longer and telemetry_AFDOGenerate has been
> failing for more than a week. Can we revert this for now?

sorry but we can't revert this. this is critical to make bisect workable.

[perezju, 2016-09-12 09:55:12]

On 2016/09/09 23:21:37, Ting-Yuan (Leo) Huang wrote:
> This has made page_cycler much longer and telemetry_AFDOGenerate has been
> failing for more than a week. Can we revert this for now?

Is there a bug tracking those issues? Maybe there are some other alternatives we
should explore to address those.

[kirr, 2016-11-03 12:13:23]

On 2016/09/12 09:55:12, perezju wrote:
> On 2016/09/09 23:21:37, Ting-Yuan (Leo) Huang wrote:
> > This has made page_cycler much longer and telemetry_AFDOGenerate has been
> > failing for more than a week. Can we revert this for now?
> 
> Is there a bug tracking those issues? Maybe there are some other alternatives
we
> should explore to address those.

Seems like results dispersion increased in many benchmarks:
https://chromeperf.appspot.com/report?sid=a58d71220e54f0ea0e23ca51e5ab586d1c0...

[nednguyen, 2016-11-03 12:29:20]

On 2016/11/03 12:13:23, kirr wrote:
> On 2016/09/12 09:55:12, perezju wrote:
> > On 2016/09/09 23:21:37, Ting-Yuan (Leo) Huang wrote:
> > > This has made page_cycler much longer and telemetry_AFDOGenerate has been
> > > failing for more than a week. Can we revert this for now?
> > 
> > Is there a bug tracking those issues? Maybe there are some other
alternatives
> we
> > should explore to address those.
> 
> Seems like results dispersion increased in many benchmarks:
>
https://chromeperf.appspot.com/report?sid=a58d71220e54f0ea0e23ca51e5ab586d1c0...

Yes. That's the price we have to pay for getting:
1) A more truthful version of "cold load"
2) Independence between test run --> this enables our bisect system to only
bisect on a single test in a benchmark.