Add android acceptance test to telemetry.

Add android acceptance test to telemetry.

This telemetry test closely mimics the android acceptance test for power.
It uses the same websites; sits on them for 20 seconds, scrolls down, sits 20 more seconds.

BUG=421569
Committed: https://crrev.com/9f332b87e8c1a333c04218c933f9caf0f06bd2b0
Cr-Commit-Position: refs/heads/master@{#300147}

[rnephew (Reviews Here), 2014-10-15 22:39:16]

rnephew@chromium.org changed reviewers:
+ sullivan@chromium.org, tonyg@chromium.org

[rnephew (Reviews Here), 2014-10-15 22:41:12]

I am currently using test = memory.Memory because no power measurement exists in
perf/measurements. The memory tests also get the power, so using that for now.
Will add power measurement if deemed valuable.

[tonyg, 2014-10-15 23:26:44]

On 2014/10/15 22:41:12, rnephew1 wrote:
> I am currently using test = memory.Memory because no power measurement exists
in
> perf/measurements. The memory tests also get the power, so using that for now.
> Will add power measurement if deemed valuable.

I think we should add a new measurement for two reasons:
1. We want the overall energy usage of the whole script (not just the per-page
usage).
2. The memory metrics is going to add some overhead while it collects the memory
numbers.

[tonyg, 2014-10-15 23:26:49]

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
File tools/perf/page_sets/android_acceptance.py (right):

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:19: 'ScrollAction', is_smooth=True)
I believe is_smooth turns on chrome tracing. We'll want to keep it off for some
energy benchmarks because it periodically wakes up threads to ask for their
trace data.

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:21: action_runner.ScrollPage()
I don't believe they do the scroll, right? Adding the scroll here seems like a
better test to me, but I'd like to have one that faithfully reproduces and
another that we deem better.

I'd recommend energy.top_10_mobile (with scroll, mobile UA and top_10_mobile
page set) and energy.android_acceptance (no scroll, desktop UA, URLs you have,
and as a follow-up we should get them to load from the sdcard instead of WPR).

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:37:
action_runner.WaitForElement(text='Conditions of Use')
Why are we waiting for an element? AFAICT, we should have to wait for any
condition since we're just sitting idle. And waiting involves polling which is
going to affect energy consumption.

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:78: self.AddPage(AmazonPage(self))
Is it possible to write a loop that does something like:

for url in ['url1',
            'url2',
            'url3']:
  self.AddPage(AcceptancePage(self, url))

[sullivan, 2014-10-16 14:47:30]

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
File tools/perf/page_sets/android_acceptance.py (right):

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:21: action_runner.ScrollPage()

> I'd recommend energy.top_10_mobile (with scroll, mobile UA and top_10_mobile
> page set) and energy.android_acceptance (no scroll, desktop UA, URLs you have,
> and as a follow-up we should get them to load from the sdcard instead of WPR).

Also, is it possible to structure this so we can use existing page sets like the
tab_switching benchmark does? It would be great to be able to do one-off
debugging in the future, for example running with the blank_page page set to
find unneeded work, or a polymer or service_worker page set to debug problems in
those features.

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:74: user_agent_type='desktop',
Why desktop? Does the "real" acceptance test use a desktop user agent?

[tonyg, 2014-10-16 16:22:07]

>
https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
> tools/perf/page_sets/android_acceptance.py:74: user_agent_type='desktop',
> Why desktop? Does the "real" acceptance test use a desktop user agent?

Unfortunately, yes.

[sullivan, 2014-10-16 16:24:19]

On 2014/10/16 16:22:07, tonyg wrote:
> >
>
https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
> > tools/perf/page_sets/android_acceptance.py:74: user_agent_type='desktop',
> > Why desktop? Does the "real" acceptance test use a desktop user agent?
> 
> Unfortunately, yes.

Weird. Maybe add a comment?

[tonyg, 2014-10-16 16:52:54]

On 2014/10/16 16:24:19, sullivan wrote:
> On 2014/10/16 16:22:07, tonyg wrote:
> > >
> >
>
https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
> > > tools/perf/page_sets/android_acceptance.py:74: user_agent_type='desktop',
> > > Why desktop? Does the "real" acceptance test use a desktop user agent?
> > 
> > Unfortunately, yes.
> 
> Weird. Maybe add a comment?

Good idea. In any case, that's why I'm encouraging us to do one version that
faithfully repros and a second that we think is more realistic.

[rnephew (Reviews Here), 2014-10-16 17:02:25]

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
File tools/perf/page_sets/android_acceptance.py (right):

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:19: 'ScrollAction', is_smooth=True)
On 2014/10/15 23:26:49, tonyg wrote:
> I believe is_smooth turns on chrome tracing. We'll want to keep it off for
some
> energy benchmarks because it periodically wakes up threads to ask for their
> trace data.

Done.

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:21: action_runner.ScrollPage()
On 2014/10/15 23:26:49, tonyg wrote:
> I don't believe they do the scroll, right? Adding the scroll here seems like a
> better test to me, but I'd like to have one that faithfully reproduces and
> another that we deem better.
> 
> I'd recommend energy.top_10_mobile (with scroll, mobile UA and top_10_mobile
> page set) and energy.android_acceptance (no scroll, desktop UA, URLs you have,
> and as a follow-up we should get them to load from the sdcard instead of WPR).

Done.

How do I get the websites to load to the SD card on telemetry tests? I've looked
around and can't see an obvious way to push data before the test runs.

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:37:
action_runner.WaitForElement(text='Conditions of Use')
On 2014/10/15 23:26:49, tonyg wrote:
> Why are we waiting for an element? AFAICT, we should have to wait for any
> condition since we're just sitting idle. And waiting involves polling which is
> going to affect energy consumption.

Done.

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:74: user_agent_type='desktop',
On 2014/10/16 14:47:30, sullivan wrote:
> Why desktop? Does the "real" acceptance test use a desktop user agent?

Added comment explaining why it uses desktop user agent.

https://codereview.chromium.org/654263005/diff/1/tools/perf/page_sets/android...
tools/perf/page_sets/android_acceptance.py:78: self.AddPage(AmazonPage(self))
On 2014/10/15 23:26:49, tonyg wrote:
> Is it possible to write a loop that does something like:
> 
> for url in ['url1',
>             'url2',
>             'url3']:
>   self.AddPage(AcceptancePage(self, url))

Done.

[tonyg, 2014-10-17 02:16:43]

https://codereview.chromium.org/654263005/diff/20001/tools/perf/measurements/...
File tools/perf/measurements/power.py (right):

https://codereview.chromium.org/654263005/diff/20001/tools/perf/measurements/...
tools/perf/measurements/power.py:1: # Copyright 2012 The Chromium Authors. All
rights reserved.
s/2012/2014/

https://codereview.chromium.org/654263005/diff/20001/tools/perf/page_sets/top...
File tools/perf/page_sets/top_10_mobile.py (right):

https://codereview.chromium.org/654263005/diff/20001/tools/perf/page_sets/top...
tools/perf/page_sets/top_10_mobile.py:22: def RunPowerPageInteractions(self,
action_runner):
Ned is in the process of removing multiple sets of actions from page sets. If he
hasn't converted this one already, he will soon. I think, instead, we'll need to
make a new set of pages or just simply branch this one. Anyone have a creative
idea for collecting the set of URLs we care about?

Note: if you want to just land the acceptance one, that part is ready to go and
we could do this in a follow up.

[rnephew (Reviews Here), 2014-10-17 14:30:10]

https://codereview.chromium.org/654263005/diff/20001/tools/perf/measurements/...
File tools/perf/measurements/power.py (right):

https://codereview.chromium.org/654263005/diff/20001/tools/perf/measurements/...
tools/perf/measurements/power.py:1: # Copyright 2012 The Chromium Authors. All
rights reserved.
On 2014/10/17 02:16:43, tonyg wrote:
> s/2012/2014/

Done.

https://codereview.chromium.org/654263005/diff/20001/tools/perf/page_sets/top...
File tools/perf/page_sets/top_10_mobile.py (right):

https://codereview.chromium.org/654263005/diff/20001/tools/perf/page_sets/top...
tools/perf/page_sets/top_10_mobile.py:22: def RunPowerPageInteractions(self,
action_runner):
On 2014/10/17 02:16:43, tonyg wrote:
> Ned is in the process of removing multiple sets of actions from page sets. If
he
> hasn't converted this one already, he will soon. I think, instead, we'll need
to
> make a new set of pages or just simply branch this one. Anyone have a creative
> idea for collecting the set of URLs we care about?
> 
> Note: if you want to just land the acceptance one, that part is ready to go
and
> we could do this in a follow up.

Done.

[tonyg, 2014-10-17 16:24:59]

The CQ bit was checked by tonyg@chromium.org

[tonyg, 2014-10-17 16:24:59]

lgtm

[commit-bot: I haz the power, 2014-10-17 16:25:28]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/654263005/60001

[commit-bot: I haz the power, 2014-10-17 16:34:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-10-17 16:34:25]

Try jobs failed on following builders:
  chromium_presubmit on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[tonyg, 2014-10-17 16:45:25]

On 2014/10/17 16:34:25, I haz the power (commit-bot) wrote:
> Try jobs failed on following builders:
>   chromium_presubmit on tryserver.chromium.linux
>
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

Attemping to commit hash file, but corresponding data file is not in Cloud
Storage:
/mnt/scratch0/b_used/build/slave/linux/build/src/tools/perf/page_sets/data/android_acceptance_000.wpr.sha1

Did you follow these steps:
http://www.chromium.org/developers/telemetry/upload_to_cloud_storage

[sullivan, 2014-10-17 16:46:05]

On 2014/10/17 16:34:25, I haz the power (commit-bot) wrote:
> Try jobs failed on following builders:
>   chromium_presubmit on tryserver.chromium.linux
>
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

Error message:
** Presubmit ERRORS **
Attemping to commit hash file, but corresponding data file is not in Cloud
Storage:
/mnt/scratch0/b_used/build/slave/linux/build/src/tools/perf/page_sets/data/android_acceptance_000.wpr.sha1

I think you need to follow the instructions to upload to cloud storage. git cl
presubmit should do this, but also see:
http://www.chromium.org/developers/telemetry/record_a_page_set#TOC-Upload-the...

[rnephew (Reviews Here), 2014-10-17 16:59:25]

On 2014/10/17 16:46:05, sullivan wrote:
> On 2014/10/17 16:34:25, I haz the power (commit-bot) wrote:
> > Try jobs failed on following builders:
> >   chromium_presubmit on tryserver.chromium.linux
> >
>
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)
> 
> Error message:
> ** Presubmit ERRORS **
> Attemping to commit hash file, but corresponding data file is not in Cloud
> Storage:
>
/mnt/scratch0/b_used/build/slave/linux/build/src/tools/perf/page_sets/data/android_acceptance_000.wpr.sha1
> 
> I think you need to follow the instructions to upload to cloud storage. git cl
> presubmit should do this, but also see:
>
http://www.chromium.org/developers/telemetry/record_a_page_set#TOC-Upload-the...

rnephew@rnephew0:/usr/local/google/code/clankium_working/src/tools/perf$
upload_to_google_storage.py --bucket chrome-partner-telemetry 
page_sets/data/android_acceptance_000.wpr
Main> Calculating hash for page_sets/data/android_acceptance_000.wpr...
Main> Done calculating hash for page_sets/data/android_acceptance_000.wpr.
0> File page_sets/data/android_acceptance_000.wpr already exists and MD5
matches, upload skipped
Hashing 1 files took 0.020172 seconds
Uploading took 1.079513 seconds
Success!

It seems to think its already there.

[rnephew (Reviews Here), 2014-10-17 17:01:47]

Just checked, the sha1sum and *.sh1 file match as well.

rnephew@rnephew0:/usr/local/google/code/clankium_working/src/tools/perf/page_sets/data$
sha1sum android_acceptance_000.wpr
363b6f00e117fb73fa0095c2b66754b9f4a4f9cd  android_acceptance_000.wpr

rnephew@rnephew0:/usr/local/google/code/clankium_working/src/tools/perf/page_sets/data$
cat android_acceptance_000.wpr.sha1 
363b6f00e117fb73fa0095c2b66754b9f4a4f9cd

[tonyg, 2014-10-17 17:08:24]

The CQ bit was checked by tonyg@chromium.org

[commit-bot: I haz the power, 2014-10-17 17:08:55]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/654263005/60001

[tonyg, 2014-10-17 17:09:16]

On 2014/10/17 17:01:47, rnephew1 wrote:
> Just checked, the sha1sum and *.sh1 file match as well.
> 
>
rnephew@rnephew0:/usr/local/google/code/clankium_working/src/tools/perf/page_sets/data$
> sha1sum android_acceptance_000.wpr
> 363b6f00e117fb73fa0095c2b66754b9f4a4f9cd  android_acceptance_000.wpr
> 
>
rnephew@rnephew0:/usr/local/google/code/clankium_working/src/tools/perf/page_sets/data$
> cat android_acceptance_000.wpr.sha1 
> 363b6f00e117fb73fa0095c2b66754b9f4a4f9cd

I just checked the box again to see if it was a flake. Adding aiolos@ and dtu@
who might also notice something.

[commit-bot: I haz the power, 2014-10-17 17:19:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-10-17 17:19:26]

Try jobs failed on following builders:
  chromium_presubmit on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[rnephew (Reviews Here), 2014-10-17 17:34:18]

On 2014/10/17 17:19:26, I haz the power (commit-bot) wrote:
> Try jobs failed on following builders:
>   chromium_presubmit on tryserver.chromium.linux
>
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

I'm going to try to delete them, rerecord them, and upload them.

[rnephew (Reviews Here), 2014-10-17 17:44:07]

Now record_wpr isn't working for the pageset...

./record_wpr android_acceptance
Usage: record_wpr <PageSet|Benchmark>

[aiolos (Not reviewing), 2014-10-17 17:47:19]

I'm not sure why record_wpr isn't working for you now, but the presubmit doesn't
upload to the partner bucket, which is causing the issue. Do you still have the
old recording?

[sullivan, 2014-10-17 17:47:22]

On 2014/10/17 17:44:07, rnephew1 wrote:
> Now record_wpr isn't working for the pageset...
> 
> ./record_wpr android_acceptance
> Usage: record_wpr <PageSet|Benchmark>

 ./record_wpr  android_acceptance_page_set

[aiolos (Not reviewing), 2014-10-17 17:48:14]

On 2014/10/17 17:47:19, aiolos wrote:
> I'm not sure why record_wpr isn't working for you now, but the presubmit
doesn't
> upload to the partner bucket, which is causing the issue. Do you still have
the
> old recording?

Also, if you deleted the .wpr file, but didn't delete it's entry from the .json
file, that might cause record_wpr to fail.

[aiolos (Not reviewing), 2014-10-17 17:52:05]

You can work around the presubmit upload by running the following command to
manually upload it to cloud storage:

depot_tools/upload_to_google_storage.py file_name -b bucket_name

for the partner bucket, bucket_name = chrome-partner-telemetry

[rnephew (Reviews Here), 2014-10-17 17:54:55]

On 2014/10/17 17:52:05, aiolos wrote:
> You can work around the presubmit upload by running the following command to
> manually upload it to cloud storage:
> 
> depot_tools/upload_to_google_storage.py file_name -b bucket_name
> 
> for the partner bucket, bucket_name = chrome-partner-telemetry

I already tried that, and it said it was already there.

[rnephew (Reviews Here), 2014-10-17 17:55:48]

The CQ bit was checked by rnephew@chromium.org

[commit-bot: I haz the power, 2014-10-17 17:57:19]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/654263005/80001

[commit-bot: I haz the power, 2014-10-17 18:06:52]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-10-17 18:06:52]

Try jobs failed on following builders:
  chromium_presubmit on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[rnephew (Reviews Here), 2014-10-17 18:11:46]

I could do the manual upload to cloud storage after recording the new pageset,
but its still failing the presubmit saying its not there.
Anyone else have any ideas?

[rnephew (Reviews Here), 2014-10-17 18:11:47]

I could do the manual upload to cloud storage after recording the new pageset,
but its still failing the presubmit saying its not there.
Anyone else have any ideas?

[sullivan, 2014-10-17 18:16:48]

On 2014/10/17 18:11:47, rnephew1 wrote:
> I could do the manual upload to cloud storage after recording the new pageset,
> but its still failing the presubmit saying its not there.
> Anyone else have any ideas?

Kari is working on a fix in https://codereview.chromium.org/661163002/

[aiolos (Not reviewing), 2014-10-17 18:27:24]

On 2014/10/17 18:16:48, sullivan wrote:
> On 2014/10/17 18:11:47, rnephew1 wrote:
> > I could do the manual upload to cloud storage after recording the new
pageset,
> > but its still failing the presubmit saying its not there.
> > Anyone else have any ideas?
> 
> Kari is working on a fix in https://codereview.chromium.org/661163002/

It should land shortly.

[aiolos (Not reviewing), 2014-10-17 19:44:10]

The CQ bit was checked by aiolos@chromium.org

[aiolos (Not reviewing), 2014-10-17 19:44:24]

On 2014/10/17 18:27:24, aiolos wrote:
> On 2014/10/17 18:16:48, sullivan wrote:
> > On 2014/10/17 18:11:47, rnephew1 wrote:
> > > I could do the manual upload to cloud storage after recording the new
> pageset,
> > > but its still failing the presubmit saying its not there.
> > > Anyone else have any ideas?
> > 
> > Kari is working on a fix in https://codereview.chromium.org/661163002/
> 
> It should land shortly.

It landed. Poking the commit again.

[commit-bot: I haz the power, 2014-10-17 19:45:38]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/654263005/80001

[commit-bot: I haz the power, 2014-10-17 19:51:52]

Committed patchset #5 (id:80001)

[commit-bot: I haz the power, 2014-10-17 19:52:24]

Patchset 5 (id:??) landed as
https://crrev.com/9f332b87e8c1a333c04218c933f9caf0f06bd2b0
Cr-Commit-Position: refs/heads/master@{#300147}