[Telemetry] Teach Telemetry to find and run reference builds.

[Telemetry] Teach Telemetry to find and run reference builds.

This allows users to do --browser=reference instead of looking up that long
platform-specific path name. It also causes test_runner to report reference
build commands for platforms that support it.

BUG=416705
Committed: https://crrev.com/7a06ab74d25486ac7ed79a61d7f90546ecab89fd
Cr-Commit-Position: refs/heads/master@{#296291}

[tonyg, 2014-09-23 02:21:11]

tonyg@chromium.org changed reviewers:
+ dtu@chromium.org, sullivan@chromium.org, zty@chromium.org

[zty, 2014-09-23 02:36:46]

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
File tools/telemetry/telemetry/test_runner.py (left):

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
tools/telemetry/telemetry/test_runner.py:287: '-v', '--output-format=buildbot',
name]),
Could this be
[{
  "cmd": "/path/to/run_benchmark --browser=exact ....",
  "name": "exact"
},
{
  "cmd": "/path/to/run_benchmark --browser=release ....",
  "name": "release"
}]
So buildbot does not have to struggle to figure out what name to use for these
two steps?

[dtu, 2014-09-23 02:42:43]

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
File tools/telemetry/telemetry/test_runner.py (right):

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
tools/telemetry/telemetry/test_runner.py:92: has_ref = 'reference' in
browser_finder.GetAllAvailableBrowserTypes(args)
Watch out for this call. afaik it pings Buildbot for the list of trybots, and we
don't want to do that every test run. Annie will know more.

[tonyg, 2014-09-23 16:51:34]

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
File tools/telemetry/telemetry/test_runner.py (left):

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
tools/telemetry/telemetry/test_runner.py:287: '-v', '--output-format=buildbot',
name]),
On 2014/09/23 02:36:46, zty wrote:
> Could this be
> [{
>   "cmd": "/path/to/run_benchmark --browser=exact ....",
>   "name": "exact"
> },
> {
>   "cmd": "/path/to/run_benchmark --browser=release ....",
>   "name": "release"
> }]
> So buildbot does not have to struggle to figure out what name to use for these
> two steps?

I updated the format to something that I think'll be cleaner and more flexible
all around.

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
File tools/telemetry/telemetry/test_runner.py (right):

https://codereview.chromium.org/593983002/diff/1/tools/telemetry/telemetry/te...
tools/telemetry/telemetry/test_runner.py:92: has_ref = 'reference' in
browser_finder.GetAllAvailableBrowserTypes(args)
On 2014/09/23 02:42:42, dtu wrote:
> Watch out for this call. afaik it pings Buildbot for the list of trybots, and
we
> don't want to do that every test run. Annie will know more.

I think it's protected by this:
def FindAllBrowserTypes(finder_options):
  # Listing browsers requires an http request; only do this if the user is      

  # running with browser=list or a browser=trybot-* argument.                   

  if (finder_options.browser_type and
      (finder_options.browser_type == 'list' or
       finder_options.browser_type.startswith('trybot'))):
    return _GetTrybotList()
  return []

[dtu, 2014-09-23 16:55:25]

lgtm. Note that ref builds still need the --trace-tag=_ref command-line flag

[sullivan, 2014-09-23 22:01:35]

lgtm

[tonyg, 2014-09-23 22:44:20]

The CQ bit was checked by tonyg@chromium.org

[commit-bot: I haz the power, 2014-09-23 22:45:16]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/593983002/20001

[commit-bot: I haz the power, 2014-09-23 22:52:43]

Committed patchset #2 (id:20001) as 33de25ff452f5cbdf197f7a622ddb8beda95a7b8

[commit-bot: I haz the power, 2014-09-23 22:53:53]

Patchset 2 (id:??) landed as
https://crrev.com/7a06ab74d25486ac7ed79a61d7f90546ecab89fd
Cr-Commit-Position: refs/heads/master@{#296291}