Add a Telemetry based cold startup test.

Add a Telemetry based cold startup test.

This shares most of its code with the startup_warm measurement. You
must now specify either --warm or --cold to run the new startup
measurement.

Due to splitting up the original patch for this into many pieces, some
of the earlier changes didn't all work together. There are also a
couple of fixes for those in here.

BUG=None

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=217126

[tonyg, 2013-08-08 01:16:35]

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
File tools/perf/benchmarks/startup_cold.py (right):

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
tools/perf/benchmarks/startup_cold.py:9: class StartupColdBlankPage(test.Test):
Let's just have a single benchmarks/startup.py file that has all startup related
Benchmarks in it.

https://codereview.chromium.org/22300013/diff/1/tools/perf/measurements/start...
File tools/perf/measurements/startup.py (right):

https://codereview.chromium.org/22300013/diff/1/tools/perf/measurements/start...
tools/perf/measurements/startup.py:9: class
StartupTest(page_measurement.PageMeasurement):
I think the precedent is to just call this "Startup". Any reason to add "Test".

You can update the old_benchmark_names map in tools/perf/run_measurement so that
the perf bots will continue to be happy regardless of what you rename this.

https://codereview.chromium.org/22300013/diff/1/tools/perf/measurements/start...
tools/perf/measurements/startup.py:27: self._warm = False
Let's use one bool so these can't get set improperly.

https://codereview.chromium.org/22300013/diff/1/tools/telemetry/telemetry/cor...
File tools/telemetry/telemetry/core/platform/desktop_platform_backend.py
(right):

https://codereview.chromium.org/22300013/diff/1/tools/telemetry/telemetry/cor...
tools/telemetry/telemetry/core/platform/desktop_platform_backend.py:50: return
Should this be an assert?

[tonyg, 2013-08-08 01:20:50]

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
File tools/perf/benchmarks/startup_warm.py (right):

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
tools/perf/benchmarks/startup_warm.py:12: options = {'warm': True}
Oh, and I think we also need to pass --page-repeat=20 here, right?

[James Simonsen, 2013-08-08 02:06:07]

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
File tools/perf/benchmarks/startup_cold.py (right):

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
tools/perf/benchmarks/startup_cold.py:9: class StartupColdBlankPage(test.Test):
On 2013/08/08 01:16:35, tonyg wrote:
> Let's just have a single benchmarks/startup.py file that has all startup
related
> Benchmarks in it.

Done.

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
File tools/perf/benchmarks/startup_warm.py (right):

https://codereview.chromium.org/22300013/diff/1/tools/perf/benchmarks/startup...
tools/perf/benchmarks/startup_warm.py:12: options = {'warm': True}
On 2013/08/08 01:20:50, tonyg wrote:
> Oh, and I think we also need to pass --page-repeat=20 here, right?

Done.

https://codereview.chromium.org/22300013/diff/1/tools/perf/measurements/start...
File tools/perf/measurements/startup.py (right):

https://codereview.chromium.org/22300013/diff/1/tools/perf/measurements/start...
tools/perf/measurements/startup.py:9: class
StartupTest(page_measurement.PageMeasurement):
On 2013/08/08 01:16:35, tonyg wrote:
> I think the precedent is to just call this "Startup". Any reason to add
"Test".
> 
> You can update the old_benchmark_names map in tools/perf/run_measurement so
that
> the perf bots will continue to be happy regardless of what you rename this.

Done.

https://codereview.chromium.org/22300013/diff/1/tools/perf/measurements/start...
tools/perf/measurements/startup.py:27: self._warm = False
On 2013/08/08 01:16:35, tonyg wrote:
> Let's use one bool so these can't get set improperly.

Done.

https://codereview.chromium.org/22300013/diff/1/tools/telemetry/telemetry/cor...
File tools/telemetry/telemetry/core/platform/desktop_platform_backend.py
(right):

https://codereview.chromium.org/22300013/diff/1/tools/telemetry/telemetry/cor...
tools/telemetry/telemetry/core/platform/desktop_platform_backend.py:50: return
On 2013/08/08 01:16:35, tonyg wrote:
> Should this be an assert?

No. In the case of an empty profile, there's nothing to flush.

[jeremy, 2013-08-08 13:00:28]

LGTM woohoo!

https://codereview.chromium.org/22300013/diff/8001/tools/perf/benchmarks/star...
File tools/perf/benchmarks/startup.py (right):

https://codereview.chromium.org/22300013/diff/8001/tools/perf/benchmarks/star...
tools/perf/benchmarks/startup.py:13: 'pageset_repeat_iters': 5}
Are we OK with 5 iterations on all platforms?  Android said things were slow on
their end (albeit that was when things were wired up for 20 iterations).

[tonyg, 2013-08-08 15:33:11]

lgtm

https://codereview.chromium.org/22300013/diff/8001/tools/perf/benchmarks/star...
File tools/perf/benchmarks/startup.py (right):

https://codereview.chromium.org/22300013/diff/8001/tools/perf/benchmarks/star...
tools/perf/benchmarks/startup.py:20: 'pageset_repeat_iters': 5}
Any idea what this will do to variance? It is currently running with 20
iterations.

[James Simonsen, 2013-08-08 17:34:40]

https://codereview.chromium.org/22300013/diff/8001/tools/perf/benchmarks/star...
File tools/perf/benchmarks/startup.py (right):

https://codereview.chromium.org/22300013/diff/8001/tools/perf/benchmarks/star...
tools/perf/benchmarks/startup.py:13: 'pageset_repeat_iters': 5}
On 2013/08/08 13:00:28, jeremy wrote:
> Are we OK with 5 iterations on all platforms?  Android said things were slow
on
> their end (albeit that was when things were wired up for 20 iterations).

I think 5 is a good starting point for the cold test. It's significantly slower
than warm. Clearing the cache takes time, then the startup is a lot slower too.
I say we start here and change it depending on how it goes.

https://codereview.chromium.org/22300013/diff/8001/tools/perf/benchmarks/star...
tools/perf/benchmarks/startup.py:20: 'pageset_repeat_iters': 5}
On 2013/08/08 15:33:11, tonyg wrote:
> Any idea what this will do to variance? It is currently running with 20
> iterations.

Done.

[commit-bot: I haz the power, 2013-08-12 20:14:43]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/simonjam@chromium.org/22300013/20001

[commit-bot: I haz the power, 2013-08-13 00:06:16]

Change committed as 217126