OLD | NEW |
1 # Running Benchmarks | 1 # Running Benchmarks |
2 | 2 |
3 There are two entry points for running benchmarks: | 3 There are two entry points for running benchmarks: |
4 * **main.dart** - a general Dart application for running performance benchmarks | 4 * **main.dart** - a general Dart application for running performance benchmarks |
5 * **local_runner.dart** - an example Dart application | 5 * **local_runner.dart** - an example Dart application |
6 which sets up the local environment | 6 which sets up the local environment |
7 and then calls main.dart to run performance benchmarks | 7 and then calls main.dart to run performance benchmarks |
8 | 8 |
9 ## local_runner.dart | 9 ## local_runner.dart |
10 | 10 |
11 This Dart application is one example for running performance benchmarks. | 11 This Dart application is one example for running performance benchmarks. |
12 When run, this application 1) extracts a branch from a git repository | 12 When run, this application 1) extracts a branch from a git repository |
13 into a temporary directory, and 2) creates a symlink to the out or xcodebuild | 13 into a temporary directory, and 2) creates a symlink to the out or xcodebuild |
14 directory for proper package-root package resolution. | 14 directory for proper package-root package resolution. |
15 Once setup is complete, this applications calls main.dart | 15 Once setup is complete, this applications calls main.dart |
16 | 16 |
17 The required command line arguments are | 17 The required command line arguments are |
18 * **gitDir** = a path to the git repository containing the initial target source | 18 * **gitDir** = a path to the git repository containing the initial target source |
19 * **branch** = the branch containing the initial target source | 19 * **branch** = the branch containing the initial target source |
20 * **inputFile** = the instrumentation or log file | 20 * **inputFile** = the instrumentation or log file |
21 | 21 |
22 Additional arguments are passed directly to main.dart. | 22 Additional arguments are passed directly to main.dart. |
23 For example, you may want to specify --newTaskModel to measure performance | 23 For example, if the log was recorded on one machine and is played back on anothe
r, |
24 with the new task model versus the old task model, | |
25 or if the log was recorded on one machine and is played back on another, | |
26 then you might need to specify -m<oldSrcPath>,<newSrcPath> | 24 then you might need to specify -m<oldSrcPath>,<newSrcPath> |
27 to map the source paths for playback. | 25 to map the source paths for playback. |
28 When specifying additional arguments, any occurrences of @tmpSrcDir@ | 26 When specifying additional arguments, any occurrences of @tmpSrcDir@ |
29 will be replaced with the absolute path of the temporary directory | 27 will be replaced with the absolute path of the temporary directory |
30 into which the source was extracted. | 28 into which the source was extracted. |
31 | 29 |
32 ## main.dart | 30 ## main.dart |
33 | 31 |
34 This Dart application reads an instrumentation or local log file produced by | 32 This Dart application reads an instrumentation or local log file produced by |
35 analysis server, "replays" that interaction with the analysis server, | 33 analysis server, "replays" that interaction with the analysis server, |
36 compares the notifications and responses with what was recorded in the log, | 34 compares the notifications and responses with what was recorded in the log, |
37 and produces a report. It assumes that the environment for playback has | 35 and produces a report. It assumes that the environment for playback has |
38 already been setup. | 36 already been setup. |
39 The required command line arguments are | 37 The required command line arguments are |
40 * **-i, --input <filePath>** | 38 * **-i, --input <filePath>** |
41 The input file specifying how this client should interact with the server. | 39 The input file specifying how this client should interact with the server. |
42 If the input file name is "stdin", then the instructions are read from stdin. | 40 If the input file name is "stdin", then the instructions are read from stdin. |
43 * **-m, --map <oldSrcPath>,<newSrcPath>** | 41 * **-m, --map <oldSrcPath>,<newSrcPath>** |
44 This option defines a mapping from the original source directory <oldSrcPath> | 42 This option defines a mapping from the original source directory <oldSrcPath> |
45 when the instrumentation or log file was generated | 43 when the instrumentation or log file was generated |
46 to the target source directory <newSrcPath> used during performance testing. | 44 to the target source directory <newSrcPath> used during performance testing. |
47 Multiple mappings can be specified. | 45 Multiple mappings can be specified. |
48 WARNING: The contents of the target directory will be modified | 46 WARNING: The contents of the target directory will be modified |
49 * **-t, --tmpSrcDir <dirPath>** | 47 * **-t, --tmpSrcDir <dirPath>** |
50 The temporary directory containing source used during performance measurement. | 48 The temporary directory containing source used during performance measurement. |
51 WARNING: The contents of the target directory will be modified | 49 WARNING: The contents of the target directory will be modified |
52 * **--newTaskModel** enable the use of the new task model | |
53 * **-d, --diagnosticPort** localhost port on which server | 50 * **-d, --diagnosticPort** localhost port on which server |
54 will provide diagnostic web pages | 51 will provide diagnostic web pages |
55 * **-v, --verbose** Verbose logging | 52 * **-v, --verbose** Verbose logging |
56 * **--vv** Extra verbose logging | 53 * **--vv** Extra verbose logging |
57 * **-h, --help** Print this help information | 54 * **-h, --help** Print this help information |
58 | 55 |
59 For each request recorded in the input file, | 56 For each request recorded in the input file, |
60 the application sends a corresponding request to the analysis server | 57 the application sends a corresponding request to the analysis server |
61 and waits up to 60 seconds for a response to that request. | 58 and waits up to 60 seconds for a response to that request. |
62 If a response in not received in that time, then the application exits. | 59 If a response in not received in that time, then the application exits. |
63 Any responses that are received are compared with the recorded response. | 60 Any responses that are received are compared with the recorded response. |
64 | 61 |
65 For each analysis-complete notification recorded in the input file, | 62 For each analysis-complete notification recorded in the input file, |
66 the application waits for the corresponding analysis-complete notification | 63 the application waits for the corresponding analysis-complete notification |
67 from the running analysis server. | 64 from the running analysis server. |
68 While it is waiting for an analysis-complete notification, | 65 While it is waiting for an analysis-complete notification, |
69 the application monitors the stream of notifications. | 66 the application monitors the stream of notifications. |
70 If there is a period of more than 60 seconds during which no communication | 67 If there is a period of more than 60 seconds during which no communication |
71 is received from the server, the application assumes that the server is hung | 68 is received from the server, the application assumes that the server is hung |
72 and exits. | 69 and exits. |
OLD | NEW |