| OLD | NEW |
| 1 # mojo_benchmark | 1 # mojo_benchmark |
| 2 | 2 |
| 3 `mojo_benchmark` allows you to run performance tests for any Mojo application | 3 `mojo_benchmark` allows you to run performance tests for any Mojo application |
| 4 that participates in the [tracing | 4 that participates in the [tracing |
| 5 ecosystem](https://github.com/domokit/mojo/blob/master/mojo/services/tracing/int
erfaces/tracing.mojom) | 5 ecosystem](https://github.com/domokit/mojo/blob/master/mojo/services/tracing/int
erfaces/tracing.mojom) |
| 6 with no app changes required. | 6 with no app changes required. |
| 7 | 7 |
| 8 The script reads a list of benchmarks to run from a file, runs each benchmark in | 8 The script reads a list of benchmarks to run from a file, runs each benchmark in |
| 9 controlled caching conditions with tracing enabled and performs specified | 9 controlled caching conditions with tracing enabled and performs specified |
| 10 measurements on the collected trace data. | 10 measurements on the collected trace data. |
| 11 | 11 |
| 12 ## Defining benchmarks | 12 ## Defining benchmarks |
| 13 | 13 |
| 14 `mojo_benchmark` runs performance tests defined in a benchmark file. The | 14 `mojo_benchmark` runs performance tests defined in a benchmark file. The |
| 15 benchmark file is a Python dictionary of the following format: | 15 benchmark file is a Python program setting a dictionary of the following format: |
| 16 | 16 |
| 17 ```python | 17 ```python |
| 18 benchmarks = [ | 18 benchmarks = [ |
| 19 { | 19 { |
| 20 'name': '<name of the benchmark>', | 20 'name': '<name of the benchmark>', |
| 21 'app': '<url of the app to benchmark>', | 21 'app': '<url of the app to benchmark>', |
| 22 'shell-args': [], | 22 'shell-args': [], |
| 23 'duration': <duration in seconds>, | 23 'duration': <duration in seconds>, |
| 24 | 24 |
| 25 # List of measurements to make. | 25 # List of measurements to make. |
| 26 'measurements': [ | 26 'measurements': [ |
| 27 '<measurement type>/<event category>/<event name>', | 27 { |
| 28 'name': <my_measurement>, |
| 29 'spec': <spec>, |
| 30 }, |
| 31 (...) |
| 28 ], | 32 ], |
| 29 }, | 33 }, |
| 30 ] | 34 ] |
| 31 ``` | 35 ``` |
| 32 | 36 |
| 37 The benchmark file may reference the `target_os` global that will be any of |
| 38 ('android', 'linux'), indicating the system on which the benchmarks are run. |
| 39 |
| 40 ### Measurement specs |
| 41 |
| 33 The following types of measurements are available: | 42 The following types of measurements are available: |
| 34 | 43 |
| 35 - `time_until` - measures time until the first occurence of the specified event | 44 - `time_until` |
| 36 - `avg_duration` - measures the average duration of all instances of the | 45 - `time_between` |
| 37 specified event | 46 - `avg_duration` |
| 47 - `percentile_duration` |
| 48 |
| 49 `time_until` records the time until the first occurence of the targeted event. |
| 50 The underlying benchmark runner records the time origin just before issuing the |
| 51 connection call to the application being benchmarked. Results of `time_until` |
| 52 measurements are relative to this time. Spec format: |
| 53 |
| 54 ``` |
| 55 'time_until/<category>/<event>' |
| 56 ``` |
| 57 |
| 58 `time_between` records the time between the first occurence of the first |
| 59 targeted event and the first occurence of the second targeted event. Spec |
| 60 format: |
| 61 |
| 62 ``` |
| 63 'time_between/<category1>/<event1>/<category2>/<event2>' |
| 64 ``` |
| 65 |
| 66 `avg_duration` records the average duration of all occurences of the targeted |
| 67 event. Spec format: |
| 68 |
| 69 ``` |
| 70 'avg_duration/<category>/<event>' |
| 71 ``` |
| 72 |
| 73 `percentile_duration` records the value at the given percentile of durations of |
| 74 all occurences of the targeted event. Spec format: |
| 75 |
| 76 ``` |
| 77 'percentile_duration/<category>/<event>/<percentile>' |
| 78 ``` |
| 79 |
| 80 where `<percentile>` is a number between 0.0 and 0.1. |
| 38 | 81 |
| 39 ## Caching | 82 ## Caching |
| 40 | 83 |
| 41 The script runs each benchmark twice. The first run (**cold start**) clears | 84 The script runs each benchmark twice. The first run (**cold start**) clears |
| 42 caches of the following apps on startup: | 85 caches of the following apps on startup: |
| 43 | 86 |
| 44 - network_service.mojo | 87 - `network_service.mojo` |
| 45 - url_response_disk_cache.mojo | 88 - `url_response_disk_cache.mojo` |
| 46 | 89 |
| 47 The second run (**warm start**) runs immediately afterwards, without clearing | 90 The second run (**warm start**) runs immediately afterwards, without clearing |
| 48 any caches. | 91 any caches. |
| 49 | 92 |
| 50 ## Time origin | |
| 51 | |
| 52 The underlying benchmark runner records the time origin just before issuing the | |
| 53 connection call to the application being benchmarked. Results of `time_until` | |
| 54 measurements are relative to this time. | |
| 55 | |
| 56 ## Example | 93 ## Example |
| 57 | 94 |
| 58 For an app that records a trace event named "initialized" in category "my_app" | 95 For an app that records a trace event named "initialized" in category "my_app" |
| 59 once its initialization is complete, we can benchmark the initialization time of | 96 once its initialization is complete, we can benchmark the initialization time of |
| 60 the app (from the moment someone tries to connect to it to the app completing | 97 the app (from the moment someone tries to connect to it to the app completing |
| 61 its initialization) using the following benchmark file: | 98 its initialization) using the following benchmark file: |
| 62 | 99 |
| 63 ```python | 100 ```python |
| 64 benchmarks = [ | 101 benchmarks = [ |
| 65 { | 102 { |
| (...skipping 27 matching lines...) Expand all Loading... |
| 93 | 130 |
| 94 If no `--server-url` is specified, the script assumes that a local instance of | 131 If no `--server-url` is specified, the script assumes that a local instance of |
| 95 the dashboard is running at `http://localhost:8080`. The script assumes that the | 132 the dashboard is running at `http://localhost:8080`. The script assumes that the |
| 96 working directory from which it is called is a git repository and queries it to | 133 working directory from which it is called is a git repository and queries it to |
| 97 determine the sequential number identifying the revision (as the number of | 134 determine the sequential number identifying the revision (as the number of |
| 98 commits in the current branch in the repository). | 135 commits in the current branch in the repository). |
| 99 | 136 |
| 100 For more information refer to: | 137 For more information refer to: |
| 101 | 138 |
| 102 - [Catapult project](https://github.com/catapult-project/catapult) | 139 - [Catapult project](https://github.com/catapult-project/catapult) |
| OLD | NEW |