Android: allow a --single-step for test_runner.py perf.

Android: allow a --single-step for test_runner.py perf.

This is the src/ part of trying to make android up and downstream run more or
less the same stack for perf tests.
Instead of requiring all steps listed (which allows sharding), this instead
allows running a single step, with retries and other niceties such as adb
restarting, device fail over, etc..

BUG=

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=234493

[bulach, 2013-11-08 14:38:44]

hey guys,

this is a different try from https://codereview.chromium.org/28933002/

Downstream and upstream would still be different:
- there will be no sharding
- this code path would only ever be exercised in chromium.perf, whilst
downstream uses the one with the json config.

However, hopefully the change in the bots will be only in telemetry.py to
"squeeze" this command in between runtests.py and run_benchmark, and we'll be
able to share a bit more of the workarounds we're sprinkling all over the
place...

please let me know your thoughts! I'll send the telemetry.py in a moment..

[tonyg, 2013-11-08 15:26:29]

lgtm

I love this new approach. It gets us incrementally closer to sharing code
without diverging platforms upstream.

One thing I'm curious about, though, why introduce a --single-step arg instead
of checking len(steps) == 1?

[bulach, 2013-11-08 15:29:26]

On 2013/11/08 15:26:29, tonyg wrote:
> lgtm
> 
> I love this new approach. It gets us incrementally closer to sharing code
> without diverging platforms upstream.
> 
> One thing I'm curious about, though, why introduce a --single-step arg instead
> of checking len(steps) == 1?

oh, --steps is the json file name, not its contents... 
I could fallback and treat it as a command if not os.path.exists, but I feel
it's going to be dirty? :)
I'd rather have it explicit if that's ok for you?

[tonyg, 2013-11-08 15:33:31]

On 2013/11/08 15:29:26, bulach wrote:
> On 2013/11/08 15:26:29, tonyg wrote:
> > lgtm
> > 
> > I love this new approach. It gets us incrementally closer to sharing code
> > without diverging platforms upstream.
> > 
> > One thing I'm curious about, though, why introduce a --single-step arg
instead
> > of checking len(steps) == 1?
> 
> oh, --steps is the json file name, not its contents... 
> I could fallback and treat it as a command if not os.path.exists, but I feel
> it's going to be dirty? :)
> I'd rather have it explicit if that's ok for you?

sgtm

[frankf, 2013-11-08 18:56:10]

lgtm with nits and a question: Why did we decide not to shard upstream? Or is
this the first step to converge upstream/downstream?

https://codereview.chromium.org/66373004/diff/30001/build/android/pylib/perf/...
File build/android/pylib/perf/setup.py (right):

https://codereview.chromium.org/66373004/diff/30001/build/android/pylib/perf/...
build/android/pylib/perf/setup.py:70: if test_options.steps:
it's better to do this validation this in test_runner.py

https://codereview.chromium.org/66373004/diff/30001/build/android/test_runner.py
File build/android/test_runner.py (right):

https://codereview.chromium.org/66373004/diff/30001/build/android/test_runner...
build/android/test_runner.py:463: 'for the "most successful" round.')
Please move this before --steps option above. Also in the help, please make the
distinction between a JSON file versus the actual command line given here.

[frankf, 2013-11-08 19:03:41]

https://codereview.chromium.org/66373004/diff/30001/build/android/test_runner.py
File build/android/test_runner.py (right):

https://codereview.chromium.org/66373004/diff/30001/build/android/test_runner...
build/android/test_runner.py:437: option_parser.example = ('%prog perf --steps
perf_steps.json')
Also might want to add this use case to examples

[bulach, 2013-11-11 10:15:48]

thanks frank! nits addressed, CQing.

as for the question about sharding: the other patch was considered too
intrusive.. :( 
this one closes (a tiny bit) the gap between down and upstream, without
diverging (much) android versus the other platforms..

https://codereview.chromium.org/66373004/diff/30001/build/android/pylib/perf/...
File build/android/pylib/perf/setup.py (right):

https://codereview.chromium.org/66373004/diff/30001/build/android/pylib/perf/...
build/android/pylib/perf/setup.py:70: if test_options.steps:
On 2013/11/08 18:56:10, frankf wrote:
> it's better to do this validation this in test_runner.py

Done.

https://codereview.chromium.org/66373004/diff/30001/build/android/test_runner.py
File build/android/test_runner.py (right):

https://codereview.chromium.org/66373004/diff/30001/build/android/test_runner...
build/android/test_runner.py:437: option_parser.example = ('%prog perf --steps
perf_steps.json')
On 2013/11/08 19:03:42, frankf wrote:
> Also might want to add this use case to examples

Done.

https://codereview.chromium.org/66373004/diff/30001/build/android/test_runner...
build/android/test_runner.py:463: 'for the "most successful" round.')
On 2013/11/08 18:56:10, frankf wrote:
> Please move this before --steps option above. Also in the help, please make
the
> distinction between a JSON file versus the actual command line given here.

Done.

[commit-bot: I haz the power, 2013-11-11 10:16:55]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/bulach@chromium.org/66373004/120001

[commit-bot: I haz the power, 2013-11-11 10:30:40]

Retried try job too often on chromium_presubmit for step(s) presubmit
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=chromium_p...

[commit-bot: I haz the power, 2013-11-11 11:20:38]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/bulach@chromium.org/66373004/120001

[commit-bot: I haz the power, 2013-11-11 11:33:18]

Retried try job too often on chromium_presubmit for step(s) presubmit
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=chromium_p...

[commit-bot: I haz the power, 2013-11-12 09:26:11]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/bulach@chromium.org/66373004/120001

[commit-bot: I haz the power, 2013-11-12 09:49:30]

Step "update" is always a major failure.
Look at the try server FAQ for more details.
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=android_db...

[commit-bot: I haz the power, 2013-11-12 11:47:36]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/bulach@chromium.org/66373004/120001

[commit-bot: I haz the power, 2013-11-12 12:59:54]

Change committed as 234493