OLD | NEW |
1 # Copyright (c) 2013 The Chromium Authors. All rights reserved. | 1 # Copyright (c) 2013 The Chromium Authors. All rights reserved. |
2 # Use of this source code is governed by a BSD-style license that can be | 2 # Use of this source code is governed by a BSD-style license that can be |
3 # found in the LICENSE file. | 3 # found in the LICENSE file. |
4 | 4 |
5 """Config file for Run Performance Test Bisect Tool | 5 """Config file read by run-bisect-perf-regression.py. |
6 | 6 |
7 This script is intended for use by anyone that wants to run a remote bisection | 7 This script is intended for use by anyone that wants to run a remote bisection |
8 on a range of revisions to look for a performance regression. Modify the config | 8 on a range of revisions to look for a performance regression. Modify the config |
9 below and add the revision range, performance command, and metric. You can then | 9 below and add the revision range, performance command, and metric. You can then |
10 run a git try <bot>. | 10 run a git try <bot>. |
11 | 11 |
12 Changes to this file should never be submitted. | 12 Changes to this file should never be submitted. |
13 | 13 |
14 Args: | 14 Args: |
15 'command': This is the full command line to pass to the | 15 'command': This is the full command to execute the test. |
16 bisect-perf-regression.py script in order to execute the test. | |
17 'good_revision': An svn or git revision where the metric hadn't regressed yet. | 16 'good_revision': An svn or git revision where the metric hadn't regressed yet. |
18 'bad_revision': An svn or git revision sometime after the metric had | 17 'bad_revision': An svn or git revision sometime after the metric regressed. |
19 regressed. | |
20 'metric': The name of the metric to parse out from the results of the | 18 'metric': The name of the metric to parse out from the results of the |
21 performance test. You can retrieve the metric by looking at the stdio of | 19 performance test. You can retrieve the metric by looking at the stdio of |
22 the performance test. Look for lines of the format: | 20 the performance test. Look for lines of the format: |
23 | 21 RESULT <graph>: <trace>= <value> <units> |
24 RESULT <graph>: <trace>= <value> <units> | |
25 | |
26 The metric name is "<graph>/<trace>". | 22 The metric name is "<graph>/<trace>". |
27 'repeat_count': The number of times to repeat the performance test. | 23 'repeat_count': The number of times to repeat the performance test. |
28 'max_time_minutes': The script will attempt to run the performance test | 24 'max_time_minutes': The script will attempt to run the performance test |
29 "repeat_count" times, unless it exceeds "max_time_minutes". | 25 "repeat_count" times, unless it exceeds "max_time_minutes". |
30 'truncate_percent': Discard the highest/lowest % values from performance test. | 26 'truncate_percent': The highest/lowest % values will be discarded before |
| 27 computing the mean result for each revision. |
31 | 28 |
32 Sample config: | 29 Sample config: |
33 | 30 |
34 config = { | 31 config = { |
35 'command': './tools/perf/run_measurement --browser=release blink_perf third_pa
rty/WebKit/PerformanceTests/Layout/floats_50_100.html', | 32 'command': './tools/perf/run_benchmark --browser=release sunspider', |
36 'good_revision': '233015', | 33 'metric': 'Total/Total', |
37 'bad_revision': '233115', | 34 'good_revision': '14ac2486c0eba1266d2da1c52e8759d9c784fe80', |
38 'metric': 'floats_50_100/floats_50_100', | 35 'bad_revision': 'fcf8643d31301eea990a4c42d7d8c9fc30cc33ec', |
39 'repeat_count': '20', | 36 'repeat_count': '20', |
40 'max_time_minutes': '20', | 37 'max_time_minutes': '20', |
41 'truncate_percent': '25', | 38 'truncate_percent': '25', |
42 } | 39 } |
43 | 40 |
44 On Windows: | 41 For Windows, if you're calling a python script you will need to add "python" |
45 - If you're calling a python script you will need to add "python" to | 42 to the command, so the command would be changed to: |
46 the command: | 43 'python tools/perf/run_benchmark -v --browser=release sunspider', |
| 44 |
| 45 For ChromeOS: |
| 46 - For good and bad revision, the script may accept either ChromeOS versions |
| 47 or unix timestamps. |
| 48 - You don't need to specify --identity and --remote, they will be added to |
| 49 the command using the bot's BISECT_CROS_IP and BISECT_CROS_BOARD values |
| 50 - Example: |
47 | 51 |
48 config = { | 52 config = { |
49 'command': 'python tools/perf/run_measurement -v --browser=release kraken', | 53 'command': ('./tools/perf/run_measurement -v --browser=cros-chrome-guest ' |
50 'good_revision': '185319', | 54 'dromaeo.jslibstylejquery') |
51 'bad_revision': '185364', | |
52 'metric': 'Total/Total', | |
53 'repeat_count': '20', | |
54 'max_time_minutes': '20', | |
55 'truncate_percent': '25', | |
56 } | |
57 | |
58 | |
59 On ChromeOS: | |
60 - Script accepts either ChromeOS versions, or unix timestamps as revisions. | |
61 - You don't need to specify --identity and --remote, they will be added to | |
62 the command using the bot's BISECT_CROS_IP and BISECT_CROS_BOARD values. | |
63 | |
64 config = { | |
65 'command': './tools/perf/run_measurement -v '\ | |
66 '--browser=cros-chrome-guest '\ | |
67 'dromaeo tools/perf/page_sets/dromaeo/jslibstylejquery.json', | |
68 'good_revision': '4086.0.0', | 55 'good_revision': '4086.0.0', |
69 'bad_revision': '4087.0.0', | 56 'bad_revision': '4087.0.0', |
70 'metric': 'jslib/jslib', | 57 'metric': 'jslib/jslib', |
71 'repeat_count': '20', | |
72 'max_time_minutes': '20', | |
73 'truncate_percent': '25', | |
74 } | 58 } |
75 | |
76 """ | 59 """ |
77 | 60 |
78 config = { | 61 config = { |
79 'command': '', | 62 'command': '', |
80 'good_revision': '', | 63 'good_revision': '', |
81 'bad_revision': '', | 64 'bad_revision': '', |
82 'metric': '', | 65 'metric': '', |
83 'repeat_count':'', | 66 'repeat_count':'', |
84 'max_time_minutes': '', | 67 'max_time_minutes': '', |
85 'truncate_percent':'', | 68 'truncate_percent':'', |
86 } | 69 } |
87 | 70 |
88 # Workaround git try issue, see crbug.com/257689 | 71 # Workaround git try issue, see crbug.com/257689 |
OLD | NEW |