Re-enable tab switching flash energy tests.

Re-enable tab switching flash energy tests.

Also add a version that runs plugin power saver.

We'd like to compare a set of flash-heavy pages with or without
plugin power saver.

R=sullivan@chromium.org
BUG=478918
Committed: https://crrev.com/c811ac094d2d64f85989a19d5743c324c49e1e3a
Cr-Commit-Position: refs/heads/master@{#329314}

[Dan Beam, 2015-05-08 01:16:21]

https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
File tools/perf/benchmarks/tab_switching.py (right):

https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
tools/perf/benchmarks/tab_switching.py:117: class
TabSwitchingPluginPowerSaver(benchmark.Benchmark):
could also derive TabSwitchingFlashEnergyCases if you want...

[achuithb, 2015-05-08 05:08:59]

achuith@chromium.org changed reviewers:
+ nednguyen@google.com

[achuithb, 2015-05-08 05:09:00]

Ned is a better reviewer for this

[sullivan, 2015-05-08 14:09:08]

sullivan@chromium.org changed reviewers:
+ sullivan@chromium.org

[sullivan, 2015-05-08 14:09:08]

+andresantoso, who's been doing some energy work in telemetry for Mac

Adding myself as a reviewer too, since I am the original author of the test.

Some high-level comments:
1) This test didn't go through a lot of testing originally. Please watch it as
it runs and verify that flash plays correctly without plugin power saver.
2) Andre has been leaning towards one page at a time testing for our newer
desktop power benchmarks. However, my understanding is that flash uses a lot of
energy because it causes many wakeups in background tabs. Is that correct? If
so, I think this is still a good test. Andre, what do you think?
3) We like to see a clear case where a new benchmark is necessary. Have you done
local runs comparing the results of this benchmark with/without plugin power
saver enabled? (Note that the perf trybots don't have flash, so they won't give
good results. The waterfall does have flash though)

https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
File tools/perf/benchmarks/tab_switching.py (right):

https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
tools/perf/benchmarks/tab_switching.py:105: @benchmark.Enabled('has tabs')
This will run on some android bots (downstream). I think you want to explicitly
state which platforms you're testing on ('linux', 'mac', win', 'chromeos')?

[nednguyen, 2015-05-08 16:23:00]

On 2015/05/08 14:09:08, sullivan wrote:
> +andresantoso, who's been doing some energy work in telemetry for Mac
> 
> Adding myself as a reviewer too, since I am the original author of the test.
> 
> Some high-level comments:
> 1) This test didn't go through a lot of testing originally. Please watch it as
> it runs and verify that flash plays correctly without plugin power saver.
> 2) Andre has been leaning towards one page at a time testing for our newer
> desktop power benchmarks. However, my understanding is that flash uses a lot
of
> energy because it causes many wakeups in background tabs. Is that correct? If
> so, I think this is still a good test. Andre, what do you think?
> 3) We like to see a clear case where a new benchmark is necessary. Have you
done
> local runs comparing the results of this benchmark with/without plugin power
> saver enabled? (Note that the perf trybots don't have flash, so they won't
give
> good results. The waterfall does have flash though)
> 
>
https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
> File tools/perf/benchmarks/tab_switching.py (right):
> 
>
https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
> tools/perf/benchmarks/tab_switching.py:105: @benchmark.Enabled('has tabs')
> This will run on some android bots (downstream). I think you want to
explicitly
> state which platforms you're testing on ('linux', 'mac', win', 'chromeos')?

I defer reviewing this to Annie :-)

[Andre, 2015-05-08 16:31:42]

We wanted to run one page at a time for power measurement because the data is
clean for each page,
so that we can identify which pages uses more power. See power.top_25 and
power.top_10 (currently
only running on Mac). But if you want to specifically test background tabs, than
those tests won't help.

[Dan Beam, 2015-05-08 21:55:14]

dbeam@chromium.org changed reviewers:
- achuith@chromium.org, nednguyen@google.com

[Dan Beam, 2015-05-08 23:32:51]

On 2015/05/08 14:09:08, sullivan wrote:
> Adding myself as a reviewer too, since I am the original author of the test.

R= -achuith@, +sullivan@

> 1) This test didn't go through a lot of testing originally. Please watch it as
> it runs and verify that flash plays correctly without plugin power saver.

Flash wasn't working with Chromium (just hung).  I built officially and disabled
loading libpepflashplayer.so (I'm on Linux) to see cute catz being petted.

> 2) Andre has been leaning towards one page at a time testing for our newer
> desktop power benchmarks. However, my understanding is that flash uses a lot
of
> energy because it causes many wakeups in background tabs. Is that correct? If
> so, I think this is still a good test. Andre, what do you think?              
                                                   

On 2015/05/08 16:31:42, Andre wrote:
> We wanted to run one page at a time for power measurement because the data is
> clean for each page,
> so that we can identify which pages uses more power. See power.top_25 and 
> power.top_10 (currently
> only running on Mac). But if you want to specifically test background tabs,
than
> those tests won't help.

Flash runs in a separate process that's harder to control/throttle and
historically has gratuitous animation.  Anecdotally, Chrome lasts much longer
with Flash disabled.

Part of the goal of Plugin Power Saver is power parity with Safari, which
implements similar plugin throttling.  We simply want to run pages that have
lots of throttleable plugins (as marked by [1]) with and without Plugin Power
Saver to verify it actually works (we don't have a lot of hard data, mainly
intuition).

[1]
https://code.google.com/p/chromium/codesearch#chromium/src/content/renderer/p...

> 3) We like to see a clear case where a new benchmark is necessary. Have you
done
> local runs comparing the results of this benchmark with/without plugin power
> saver enabled?

I got the tests running locally but the results file:/// link doesn't look right
(lots of data fields missing...).

> (Note that the perf trybots don't have flash, so they won't give
> good results. The waterfall does have flash though)

That's good enough.

https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
File tools/perf/benchmarks/tab_switching.py (right):

https://codereview.chromium.org/1137613002/diff/1/tools/perf/benchmarks/tab_s...
tools/perf/benchmarks/tab_switching.py:105: @benchmark.Enabled('has tabs')
On 2015/05/08 14:09:08, sullivan wrote:
> This will run on some android bots (downstream). I think you want to
explicitly
> state which platforms you're testing on ('linux', 'mac', win', 'chromeos')?

Done.

[sullivan, 2015-05-11 21:00:31]

lgtm

I saw the results you linked offline, seems like PPS was a clear win locally.

[Dan Beam, 2015-05-11 21:02:25]

The CQ bit was checked by dbeam@chromium.org

[commit-bot: I haz the power, 2015-05-11 21:05:38]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1137613002/20001

[commit-bot: I haz the power, 2015-05-11 23:11:04]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-11 23:11:06]

Try jobs failed on following builders:
  linux_perf_bisect on tryserver.chromium.perf (JOB_TIMED_OUT, no build URL)
  win_perf_bisect on tryserver.chromium.perf (JOB_TIMED_OUT, no build URL)

[Dan Beam, 2015-05-11 23:13:01]

On 2015/05/11 23:11:06, I haz the power (commit-bot) wrote:
> Try jobs failed on following builders:
>   linux_perf_bisect on tryserver.chromium.perf (JOB_TIMED_OUT, no build URL)
>   win_perf_bisect on tryserver.chromium.perf (JOB_TIMED_OUT, no build URL)

hmmm, maybe temporary infra issue?  these seem to exist:

http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect
http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect

[Dan Beam, 2015-05-11 23:13:07]

The CQ bit was checked by dbeam@chromium.org

[commit-bot: I haz the power, 2015-05-11 23:14:38]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1137613002/20001

[commit-bot: I haz the power, 2015-05-12 01:17:32]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-12 01:17:33]

Try jobs failed on following builders:
  linux_perf_bisect on tryserver.chromium.perf (JOB_TIMED_OUT, no build URL)
  win_perf_bisect on tryserver.chromium.perf (JOB_TIMED_OUT, no build URL)

[Dan Beam, 2015-05-12 01:34:38]

The CQ bit was checked by dbeam@chromium.org

[commit-bot: I haz the power, 2015-05-12 01:37:51]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1137613002/20001

[commit-bot: I haz the power, 2015-05-12 02:09:43]

Committed patchset #2 (id:20001)

[commit-bot: I haz the power, 2015-05-12 02:10:31]

Patchset 2 (id:??) landed as
https://crrev.com/c811ac094d2d64f85989a19d5743c324c49e1e3a
Cr-Commit-Position: refs/heads/master@{#329314}