Add new benchmark suite runner.

Add new benchmark suite runner.

TEST=./tools/run_benchmarks.py benchmarks/v8.json
TEST=cd tools/unittests; python -m unittest run_benchmarks_test

Does not support custom results processors yet. Will implement that in a future CL.

BUG=374740
LOG=n
R=jkummerow@chromium.org

Committed: https://code.google.com/p/v8/source/detail?r=21570

[Michael Achenbach, 2014-05-22 09:21:03]

PTAL. Early stage review. Please review only documentation and script options.
Implementation details are work in progress.

[Michael Achenbach, 2014-05-22 13:57:06]

[+ Ulan]. Please have a look at the documentation and evaluate if all necessary
features are covered.

[m.m.capewell, 2014-05-22 15:27:02]

https://codereview.chromium.org/293023006/diff/40001/benchmarks/octane.json
File benchmarks/octane.json (right):

https://codereview.chromium.org/293023006/diff/40001/benchmarks/octane.json#n...
benchmarks/octane.json:4: "results_regexp": "^%s: (\\d+)$",
DBC: on really slow platforms, Octane results can be reported with a decimal
place, eg. 42.4.

[ulan, 2014-05-23 10:02:35]

https://codereview.chromium.org/293023006/diff/40001/tools/run-benchmarks.py
File tools/run-benchmarks.py (right):

https://codereview.chromium.org/293023006/diff/40001/tools/run-benchmarks.py#...
tools/run-benchmarks.py:50: "results_regexp": "^%s: (\\d+)$",
Note that some benchmarks have custom result parsers (e.g. V8GCHeapSizeKb).

https://codereview.chromium.org/293023006/diff/40001/tools/run-benchmarks.py#...
tools/run-benchmarks.py:78: Path pieces are concatenated. D8 is always run with
the suite's path as cwd.
Would be nice to the binary name (d8) as a parameter instead of hardcoding it.

[Michael Achenbach, 2014-05-23 13:57:39]

Done. Will focus on implementation now...

https://codereview.chromium.org/293023006/diff/40001/benchmarks/octane.json
File benchmarks/octane.json (right):

https://codereview.chromium.org/293023006/diff/40001/benchmarks/octane.json#n...
benchmarks/octane.json:4: "results_regexp": "^%s: (\\d+)$",
On 2014/05/22 15:27:02, m.m.capewell wrote:
> DBC: on really slow platforms, Octane results can be reported with a decimal
> place, eg. 42.4.

Good point! The values transmitted to the dashboard are strings anyway...

https://codereview.chromium.org/293023006/diff/40001/tools/run-benchmarks.py
File tools/run-benchmarks.py (right):

https://codereview.chromium.org/293023006/diff/40001/tools/run-benchmarks.py#...
tools/run-benchmarks.py:50: "results_regexp": "^%s: (\\d+)$",
On 2014/05/23 10:02:35, ulan wrote:
> Note that some benchmarks have custom result parsers (e.g. V8GCHeapSizeKb).

I added the ability for custom results processors. Existing processors might
need a small rewrite as I won't allow a results parser to be also the d8 runner.

https://codereview.chromium.org/293023006/diff/40001/tools/run-benchmarks.py#...
tools/run-benchmarks.py:78: Path pieces are concatenated. D8 is always run with
the suite's path as cwd.
On 2014/05/23 10:02:35, ulan wrote:
> Would be nice to the binary name (d8) as a parameter instead of hardcoding it.

Done. Can be configured arbitrarily now with d8 as default.

[Michael Achenbach, 2014-05-28 09:35:18]

PTAL. Implementation is ready for review. Custom results processors will be
postponed for now.

[Jakob Kummerow, 2014-05-28 11:15:05]

LGTM. I'm pretty much rubber-stamping this -- it's just a test driver; as long
as it does what you want it to do, no need to obsess over implementation details
:-)

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py
File tools/run_benchmarks.py (right):

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py...
tools/run_benchmarks.py:244: # TODO(machenbach): This requires +.exe if ran on
windows.
nit: s/ran/run/

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py...
tools/run_benchmarks.py:346: if not options.mode.lower() in ["debug", "release",
"optdebug"]:
arguably running benchmarks in anything other than release mode doesn't make
sense anyway

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py...
tools/run_benchmarks.py:387: output =
commands.Execute(runnable.GetCommand(shell_dir), timeout=60)
for slow (mobile) devices 60 seconds might be too tight

[Jakob Kummerow, 2014-05-28 11:16:27]

one more thing.

https://codereview.chromium.org/293023006/diff/230004/benchmarks/octane.json
File benchmarks/octane.json (right):

https://codereview.chromium.org/293023006/diff/230004/benchmarks/octane.json#...
benchmarks/octane.json:1: {
This file should be named "v8.json". The Octane suite is much bigger and not
checked in here.

[Michael Achenbach, 2014-05-28 12:54:49]

Done.

https://codereview.chromium.org/293023006/diff/230004/benchmarks/octane.json
File benchmarks/octane.json (right):

https://codereview.chromium.org/293023006/diff/230004/benchmarks/octane.json#...
benchmarks/octane.json:1: {
On 2014/05/28 11:16:27, Jakob wrote:
> This file should be named "v8.json". The Octane suite is much bigger and not
> checked in here.

Done.

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py
File tools/run_benchmarks.py (right):

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py...
tools/run_benchmarks.py:244: # TODO(machenbach): This requires +.exe if ran on
windows.
On 2014/05/28 11:15:05, Jakob wrote:
> nit: s/ran/run/

Done.

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py...
tools/run_benchmarks.py:346: if not options.mode.lower() in ["debug", "release",
"optdebug"]:
On 2014/05/28 11:15:05, Jakob wrote:
> arguably running benchmarks in anything other than release mode doesn't make
> sense anyway

Done. Dropped the option completely.

https://codereview.chromium.org/293023006/diff/230004/tools/run_benchmarks.py...
tools/run_benchmarks.py:387: output =
commands.Execute(runnable.GetCommand(shell_dir), timeout=60)
On 2014/05/28 11:15:05, Jakob wrote:
> for slow (mobile) devices 60 seconds might be too tight

Will make it configurable in a followup. Marked as TODO for now.

[Michael Achenbach, 2014-05-28 13:05:26]

Committed patchset #15 manually as r21570 (presubmit successful).