Telemetry: Quiescent power benchmarks for Mac

Telemetry: Quiescent power benchmarks for Mac

Add benchmark and measurement for measuring quiescent power, and enable
it on Mac.
Moved the workaround for http://crbug.com/419786 so that we don't lose idle
wakeups numbers for the renderer process.

Committed: https://crrev.com/9166080685403f1360af8e7578e3e102e513e78b
Cr-Commit-Position: refs/heads/master@{#326698}

[Andre, 2015-04-20 21:15:41]

andresantoso@chromium.org changed reviewers:
+ ccameron@chromium.org, shrike@chromium.org

[Andre, 2015-04-20 21:15:42]

This is what I'm playing around with, and can be a starting point for
discussion.
You can run it like this:
    tools/perf/run_benchmark --browser=canary energy.top_25
Is Top25PageSet a good one to use for our main benchmark?
Let me know what you guys think.

https://codereview.chromium.org/1092393002/diff/1/tools/telemetry/telemetry/c...
File tools/telemetry/telemetry/core/browser.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/telemetry/telemetry/c...
tools/telemetry/telemetry/core/browser.py:211: #   del result['Renderer']
This is causing renderer numbers to be missing. Need to find a way to fix.

[ccameron, 2015-04-20 21:36:33]

Thanks -- this looks reasonable. I'd like to dig a bit more into the renderer
issue.

https://codereview.chromium.org/1092393002/diff/1/tools/perf/benchmarks/energ...
File tools/perf/benchmarks/energy.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/perf/benchmarks/energ...
tools/perf/benchmarks/energy.py:12: """Top 25 energy test."""
I'd like to add some simpler pages, in particular
- a video
- a CSS animation
- a rAF animation
- a blinking cursor
- a blank/idle page

https://codereview.chromium.org/1092393002/diff/1/tools/perf/measurements/ene...
File tools/perf/measurements/energy.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/perf/measurements/ene...
tools/perf/measurements/energy.py:19: metrics = [
power.PowerMetric(tab.browser.platform), cpu.CpuMetric(tab.browser) ]
It may be a good idea to wait for a few seconds here, just to ensure that things
are "really" quiescent. Might be worth figuring out what that level is
empirically on some machine.

https://codereview.chromium.org/1092393002/diff/1/tools/telemetry/telemetry/c...
File tools/telemetry/telemetry/core/browser.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/telemetry/telemetry/c...
tools/telemetry/telemetry/core/browser.py:211: #   del result['Renderer']
On 2015/04/20 21:15:41, Andre wrote:
> This is causing renderer numbers to be missing. Need to find a way to fix.

Yeah, the argumentation in the bug seems pretty suspect for a test like this
(we're on the same renderer process throughout, as would be the case for any of
tests that we'd find useful).

[Andre, 2015-04-20 21:52:40]

https://codereview.chromium.org/1092393002/diff/1/tools/perf/benchmarks/energ...
File tools/perf/benchmarks/energy.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/perf/benchmarks/energ...
tools/perf/benchmarks/energy.py:12: """Top 25 energy test."""
On 2015/04/20 21:36:33, ccameron wrote:
> I'd like to add some simpler pages, in particular
> - a video
> - a CSS animation
> - a rAF animation
> - a blinking cursor
> - a blank/idle page

Sounds good. We should create a new pageset instead of adding to this one, and
then track the numbers for both pagesets.

https://codereview.chromium.org/1092393002/diff/1/tools/perf/measurements/ene...
File tools/perf/measurements/energy.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/perf/measurements/ene...
tools/perf/measurements/energy.py:19: metrics = [
power.PowerMetric(tab.browser.platform), cpu.CpuMetric(tab.browser) ]
On 2015/04/20 21:36:33, ccameron wrote:
> It may be a good idea to wait for a few seconds here, just to ensure that
things
> are "really" quiescent. Might be worth figuring out what that level is
> empirically on some machine.

metric.PowerMetric supports measuring quiescent power draw by passing a duration
parameter to the constructor (measurements.TabSwitching does this).

[Andre, 2015-04-20 23:44:43]

Possible fix for missing idle_wakeups_renderer numbers in patchset #2.
If we want cpu_utilization numbers for our benchmark, we can add it to
metrics.PowerMetric instead of using metrics.CpuMetric.

[Andre, 2015-04-21 00:00:29]

Patchset #2 (id:20001) has been deleted

[Andre, 2015-04-21 00:04:21]

andresantoso@chromium.org changed reviewers:
+ sullivan@chromium.org

[Andre, 2015-04-21 00:04:22]

sullivan@ PTAL.
ccameron, shrike, and I are starting to look into energy efficiency on Mac,
where Safari does better than Chrome in general.
First we need to add measurements. This is what we're thinking to start with,
would appreciate your input.

[sullivan, 2015-04-21 15:56:53]

+dtu, who did some work on the original power implementations, in case he has
comments on structure.

One thing I'd like to see is comments that clarify the overlap between this and
the tab_switching tests. We had been using the tab_switching tests in the past
for idle wakeups optimization because they gave us data on what was happening in
background tabs.

https://codereview.chromium.org/1092393002/diff/1/tools/perf/benchmarks/energ...
File tools/perf/benchmarks/energy.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/perf/benchmarks/energ...
tools/perf/benchmarks/energy.py:12: """Top 25 energy test."""
On 2015/04/20 21:52:39, Andre wrote:
> On 2015/04/20 21:36:33, ccameron wrote:
> > I'd like to add some simpler pages, in particular
> > - a video
> > - a CSS animation
> > - a rAF animation
> > - a blinking cursor
> > - a blank/idle page
> 
> Sounds good. We should create a new pageset instead of adding to this one, and
> then track the numbers for both pagesets.

There already exists a tough_energy_cases pageset, would probably be better to
update it instead of making a new one.

https://codereview.chromium.org/1092393002/diff/40001/tools/perf/measurements...
File tools/perf/measurements/energy.py (right):

https://codereview.chromium.org/1092393002/diff/40001/tools/perf/measurements...
tools/perf/measurements/energy.py:12: class Energy(page_test.PageTest):
It's pretty confusing to have both "Power" and "Energy" measurements/benchmarks
with no comments in either file about the reasoning for having both and the
naming. Is it possible to move these new measurements and benchmarks into the
power.py files so they're all in the same place?

[Andre, 2015-04-22 00:46:55]

Thanks!
We are looking for good data on individual pages, so that they can be analyzed
and improved. TabSwitching measures the entire set of pages at once, so it's
less useful for what we are trying to do.

https://codereview.chromium.org/1092393002/diff/1/tools/perf/measurements/ene...
File tools/perf/measurements/energy.py (right):

https://codereview.chromium.org/1092393002/diff/1/tools/perf/measurements/ene...
tools/perf/measurements/energy.py:19: metrics = [
power.PowerMetric(tab.browser.platform), cpu.CpuMetric(tab.browser) ]
On 2015/04/20 21:36:33, ccameron wrote:
> It may be a good idea to wait for a few seconds here, just to ensure that
things
> are "really" quiescent. Might be worth figuring out what that level is
> empirically on some machine.

I've added util.WaitFor(tab.hasReachedQuiescence) call here.

https://codereview.chromium.org/1092393002/diff/40001/tools/perf/measurements...
File tools/perf/measurements/energy.py (right):

https://codereview.chromium.org/1092393002/diff/40001/tools/perf/measurements...
tools/perf/measurements/energy.py:12: class Energy(page_test.PageTest):
On 2015/04/21 15:56:53, sullivan wrote:
> It's pretty confusing to have both "Power" and "Energy"
measurements/benchmarks
> with no comments in either file about the reasoning for having both and the
> naming. Is it possible to move these new measurements and benchmarks into the
> power.py files so they're all in the same place?

Done, merged into power.py.

[sullivan, 2015-04-22 13:43:33]

lgtm

Please get a green run on a mac perf trybot before submitting:
http://www.chromium.org/developers/performance-try-bots

[dtu, 2015-04-22 14:50:36]

dtu@chromium.org changed reviewers:
+ dtu@chromium.org

[dtu, 2015-04-22 14:50:37]

lgtm

[Andre, 2015-04-23 05:18:15]

Fixed page_cycler_unittest.
It was still expecting renderer cpu_utilization in results even after the
workaround for http://crbug.com/419786.

[sullivan, 2015-04-23 19:58:43]

On 2015/04/23 05:18:15, Andre wrote:
> Fixed page_cycler_unittest.
> It was still expecting renderer cpu_utilization in results even after the
> workaround for http://crbug.com/419786.

Thanks! still lgtm

[Andre, 2015-04-24 00:11:26]

The CQ bit was checked by andresantoso@chromium.org

[Andre, 2015-04-24 00:11:26]

The patchset sent to the CQ was uploaded after l-g-t-m from dtu@chromium.org
Link to the patchset: https://codereview.chromium.org/1092393002/#ps80001
(title: "Fix unit test")

[commit-bot: I haz the power, 2015-04-24 00:12:42]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1092393002/80001

[commit-bot: I haz the power, 2015-04-24 00:20:14]

Committed patchset #4 (id:80001)

[commit-bot: I haz the power, 2015-04-24 00:22:08]

Patchset 4 (id:??) landed as
https://crrev.com/9166080685403f1360af8e7578e3e102e513e78b
Cr-Commit-Position: refs/heads/master@{#326698}