entrypoint to measure performance of the analyzer frontend

entrypoint to measure performance of the analyzer frontend

R=paulberry@google.com

Committed: https://github.com/dart-lang/sdk/commit/4af56d5647cb01622c3af028a9265a98600b2384

[Siggi Cherem (dart-lang), 2016-09-15 22:06:47]

Patchset #1 (id:1) has been deleted

[Siggi Cherem (dart-lang), 2016-09-15 22:11:04]

Patchset #2 (id:40001) has been deleted

[Siggi Cherem (dart-lang), 2016-09-15 22:11:09]

Patchset #1 (id:20001) has been deleted

[Siggi Cherem (dart-lang), 2016-09-15 22:22:20]

Patchset #1 (id:60001) has been deleted

[Siggi Cherem (dart-lang), 2016-09-15 22:24:36]

Patchset #1 (id:80001) has been deleted

[Siggi Cherem (dart-lang), 2016-09-15 22:26:08]

sigmund@google.com changed reviewers:
+ asiva@google.com, paulberry@google.com

[Siggi Cherem (dart-lang), 2016-09-15 22:26:09]

Siva - here is the code I have so far to only run the analyzer's parser

Paul - let me know if this seems to be covering what we need. Also, is there a
way to scan directly from a random-access-file without loading the contents as a
string first?

[Paul Berry, 2016-09-15 22:32:50]

lgtm.  This seems like a good start.

Note that this code will run the parser with a semi-warm VM.  (It's partially
warmed up by the initial scan).  Eventually we will want to add a mode that runs
the test N times, discards the first M runs, and averages the rest (or something
like that) so that we can measure parse speed with a fully-warmed VM and compare
the difference.

[Siggi Cherem (dart-lang), 2016-09-15 22:39:48]

> Note that this code will run the parser with a semi-warm VM.  (It's partially
> warmed up by the initial scan).  

correct! if I replace 'parseDirectives' with 'parseFull' while loading the
sources, the code works the same but we end up seeing some of the effect of the
warm-up phase.

Here is a sample output of compiling dart2js sources with `parseDirectives`
during the load:

  input size: 7569880 chars
  Loader: 187 ms, 40321 chars/ms
  Scanner: 288 ms, 26216 chars/ms
  Scanner: 265 ms, 28489 chars/ms
  Parser: 757 ms, 9998 chars/ms
  Total: 1668 ms, 4535 chars/ms

Here is what happens when you use parseFull during the load:

input size: 7569880 chars
Loader: 926 ms, 8169 chars/ms     # note this is 5x bigger
Scanner: 397 ms, 19030 chars/ms
Scanner: 225 ms, 33621 chars/ms
Parser: 390 ms, 19396 chars/ms    # note this is 2x smaller
Total: 2106 ms, 3593 chars/ms

[Siggi Cherem (dart-lang), 2016-09-15 22:43:14]

Description was changed from

==========
entrypoint to measure performance of the analyzer frontend
==========

to

==========
entrypoint to measure performance of the analyzer frontend

R=paulberry@google.com

Committed:
https://github.com/dart-lang/sdk/commit/4af56d5647cb01622c3af028a9265a98600b2384
==========

[Siggi Cherem (dart-lang), 2016-09-15 22:43:16]

Committed patchset #1 (id:100001) manually as
4af56d5647cb01622c3af028a9265a98600b2384 (presubmit successful).

[Paul Berry, 2016-09-15 22:43:31]

On 2016/09/15 22:39:48, Siggi Cherem (dart-lang) wrote:
> > Note that this code will run the parser with a semi-warm VM.  (It's
partially
> > warmed up by the initial scan).  
> 
> correct! if I replace 'parseDirectives' with 'parseFull' while loading the
> sources, the code works the same but we end up seeing some of the effect of
the
> warm-up phase.
> 
> Here is a sample output of compiling dart2js sources with `parseDirectives`
> during the load:
> 
>   input size: 7569880 chars
>   Loader: 187 ms, 40321 chars/ms
>   Scanner: 288 ms, 26216 chars/ms
>   Scanner: 265 ms, 28489 chars/ms
>   Parser: 757 ms, 9998 chars/ms
>   Total: 1668 ms, 4535 chars/ms
> 
> Here is what happens when you use parseFull during the load:
> 
> input size: 7569880 chars
> Loader: 926 ms, 8169 chars/ms     # note this is 5x bigger
> Scanner: 397 ms, 19030 chars/ms
> Scanner: 225 ms, 33621 chars/ms
> Parser: 390 ms, 19396 chars/ms    # note this is 2x smaller
> Total: 2106 ms, 3593 chars/ms

Yup, not surprising.  FWIW, my general practice when trying to find "fully
warmed up performance" is to keep doubling the number of repetitions until the
total execution time is at least 10x the first execution time.  My record is 9
doublings (running the task 512x takes only ~10x longer than running it the
first time).  I suspect the difference in your case won't be anywhere near so
dramatic, since the amount of code being analyzed is pretty significant already,
so we get a lot of warmup with each iteration.

[Brian Wilkerson, 2016-09-16 15:32:02]

brianwilkerson@google.com changed reviewers:
+ brianwilkerson@google.com

[Brian Wilkerson, 2016-09-16 15:32:02]

https://codereview.chromium.org/2339803004/diff/100001/pkg/analyzer_cli/tool/...
File pkg/analyzer_cli/tool/perf.dart (right):

https://codereview.chromium.org/2339803004/diff/100001/pkg/analyzer_cli/tool/...
pkg/analyzer_cli/tool/perf.dart:129: var next = sources.resolveUri(start,
directive.uri.stringValue);
This introduces a compilation error:

ERROR: The getter 'uri' is not defined for the class 'Directive'.
([analyzer_cli] tool/perf.dart:129)

[Siggi Cherem (dart-lang), 2016-09-16 15:37:51]

On 2016/09/16 15:32:02, Brian Wilkerson wrote:
>
https://codereview.chromium.org/2339803004/diff/100001/pkg/analyzer_cli/tool/...
> File pkg/analyzer_cli/tool/perf.dart (right):
> 
>
https://codereview.chromium.org/2339803004/diff/100001/pkg/analyzer_cli/tool/...
> pkg/analyzer_cli/tool/perf.dart:129: var next = sources.resolveUri(start,
> directive.uri.stringValue);
> This introduces a compilation error:
> 
> ERROR: The getter 'uri' is not defined for the class 'Directive'.
> ([analyzer_cli] tool/perf.dart:129)

Oh, thanks I'll send a fix shortly... can't wait for the day when we get type
promotion on "if (x is! X) continue" :-).

QQ - how did you get this error? It didn't show for me when I run:

./tools/test.py -m release -c dart2analyzer -r none
pkg/analyzer_cli/tool/perf_test