OLD | NEW |
1 # mojo_benchmark | 1 # mojo_benchmark |
2 | 2 |
3 `mojo_benchmark` allows you to run performance tests for any Mojo application | 3 `mojo_benchmark` allows you to run performance tests for any Mojo application |
4 that participates in the [tracing | 4 that participates in the [tracing |
5 ecosystem](https://github.com/domokit/mojo/blob/master/mojo/services/tracing/int
erfaces/tracing.mojom) | 5 ecosystem](https://github.com/domokit/mojo/blob/master/mojo/services/tracing/int
erfaces/tracing.mojom) |
6 with no app changes required. | 6 with no app changes required. |
7 | 7 |
8 The script reads a list of benchmarks to run from a file, runs each benchmark in | 8 The script reads a list of benchmarks to run from a file, runs each benchmark in |
9 controlled caching conditions with tracing enabled and performs specified | 9 controlled caching conditions with tracing enabled and performs specified |
10 measurements on the collected trace data. | 10 measurements on the collected trace data. |
11 | 11 |
12 ## Defining benchmarks | 12 ## Defining benchmarks |
13 | 13 |
14 `mojo_benchmark` runs performance tests defined in a benchmark file. The | 14 `mojo_benchmark` runs performance tests defined in a benchmark file. The |
15 benchmark file is a Python dictionary of the following format: | 15 benchmark file is a Python program setting a dictionary of the following format: |
16 | 16 |
17 ```python | 17 ```python |
18 benchmarks = [ | 18 benchmarks = [ |
19 { | 19 { |
20 'name': '<name of the benchmark>', | 20 'name': '<name of the benchmark>', |
21 'app': '<url of the app to benchmark>', | 21 'app': '<url of the app to benchmark>', |
22 'shell-args': [], | 22 'shell-args': [], |
23 'duration': <duration in seconds>, | 23 'duration': <duration in seconds>, |
24 | 24 |
25 # List of measurements to make. | 25 # List of measurements to make. |
26 'measurements': [ | 26 'measurements': [ |
27 '<measurement type>/<event category>/<event name>', | 27 { |
| 28 'name': <my_measurement>, |
| 29 'spec': <spec>, |
| 30 }, |
| 31 (...) |
28 ], | 32 ], |
29 }, | 33 }, |
30 ] | 34 ] |
31 ``` | 35 ``` |
32 | 36 |
| 37 The benchmark file may reference the `target_os` global that will be any of |
| 38 ('android', 'linux'), indicating the system on which the benchmarks are run. |
| 39 |
| 40 ### Measurement specs |
| 41 |
33 The following types of measurements are available: | 42 The following types of measurements are available: |
34 | 43 |
35 - `time_until` - measures time until the first occurence of the specified event | 44 - `time_until` |
36 - `avg_duration` - measures the average duration of all instances of the | 45 - `time_between` |
37 specified event | 46 - `avg_duration` |
| 47 - `percentile_duration` |
| 48 |
| 49 `time_until` records the time until the first occurence of the targeted event. |
| 50 The underlying benchmark runner records the time origin just before issuing the |
| 51 connection call to the application being benchmarked. Results of `time_until` |
| 52 measurements are relative to this time. Spec format: |
| 53 |
| 54 ``` |
| 55 'time_until/<category>/<event>' |
| 56 ``` |
| 57 |
| 58 `time_between` records the time between the first occurence of the first |
| 59 targeted event and the first occurence of the second targeted event. Spec |
| 60 format: |
| 61 |
| 62 ``` |
| 63 'time_between/<category1>/<event1>/<category2>/<event2>' |
| 64 ``` |
| 65 |
| 66 `avg_duration` records the average duration of all occurences of the targeted |
| 67 event. Spec format: |
| 68 |
| 69 ``` |
| 70 'avg_duration/<category>/<event>' |
| 71 ``` |
| 72 |
| 73 `percentile_duration` records the value at the given percentile of durations of |
| 74 all occurences of the targeted event. Spec format: |
| 75 |
| 76 ``` |
| 77 'percentile_duration/<category>/<event>/<percentile>' |
| 78 ``` |
| 79 |
| 80 where `<percentile>` is a number between 0.0 and 0.1. |
38 | 81 |
39 ## Caching | 82 ## Caching |
40 | 83 |
41 The script runs each benchmark twice. The first run (**cold start**) clears | 84 The script runs each benchmark twice. The first run (**cold start**) clears |
42 caches of the following apps on startup: | 85 caches of the following apps on startup: |
43 | 86 |
44 - network_service.mojo | 87 - `network_service.mojo` |
45 - url_response_disk_cache.mojo | 88 - `url_response_disk_cache.mojo` |
46 | 89 |
47 The second run (**warm start**) runs immediately afterwards, without clearing | 90 The second run (**warm start**) runs immediately afterwards, without clearing |
48 any caches. | 91 any caches. |
49 | 92 |
50 ## Time origin | |
51 | |
52 The underlying benchmark runner records the time origin just before issuing the | |
53 connection call to the application being benchmarked. Results of `time_until` | |
54 measurements are relative to this time. | |
55 | |
56 ## Example | 93 ## Example |
57 | 94 |
58 For an app that records a trace event named "initialized" in category "my_app" | 95 For an app that records a trace event named "initialized" in category "my_app" |
59 once its initialization is complete, we can benchmark the initialization time of | 96 once its initialization is complete, we can benchmark the initialization time of |
60 the app (from the moment someone tries to connect to it to the app completing | 97 the app (from the moment someone tries to connect to it to the app completing |
61 its initialization) using the following benchmark file: | 98 its initialization) using the following benchmark file: |
62 | 99 |
63 ```python | 100 ```python |
64 benchmarks = [ | 101 benchmarks = [ |
65 { | 102 { |
(...skipping 27 matching lines...) Expand all Loading... |
93 | 130 |
94 If no `--server-url` is specified, the script assumes that a local instance of | 131 If no `--server-url` is specified, the script assumes that a local instance of |
95 the dashboard is running at `http://localhost:8080`. The script assumes that the | 132 the dashboard is running at `http://localhost:8080`. The script assumes that the |
96 working directory from which it is called is a git repository and queries it to | 133 working directory from which it is called is a git repository and queries it to |
97 determine the sequential number identifying the revision (as the number of | 134 determine the sequential number identifying the revision (as the number of |
98 commits in the current branch in the repository). | 135 commits in the current branch in the repository). |
99 | 136 |
100 For more information refer to: | 137 For more information refer to: |
101 | 138 |
102 - [Catapult project](https://github.com/catapult-project/catapult) | 139 - [Catapult project](https://github.com/catapult-project/catapult) |
OLD | NEW |