A/B fasta performance measurement harness

A/B fasta performance measurement harness

Given two separate Dart SDK repo clones, A and B,
this pkg/front_end/tool/fasta/abcompile.dart harness
alternately runs

<A>/pkg/front_end/tool/fasta/compile.dart
then
<B>/pkg/front_end/tool/fasta/compile.dart

so that a performance comparison can be made of the difference
between fasta in A vs B.
For example, from the <A> root directory:

xcodebuild/ReleaseX64/dart
  -DbRoot=/path/to/2nd/dart/sdk/repo
  -Diterations=15
  pkg/front_end/tool/fasta/abcompile.dart
  --compile-sdk=xcodebuild/ReleaseX64/patched_sdk/
  pkg/compiler/lib/src/dart2js.dart
  --verbose

The output contains the raw data along with a summary
of the average difference, and is formatted in a way
that can be easily pasted into Google Sheets
for further analysis.

Future improvements to this include:

* Add JSON output to pkg/front_end/tool/fasta/compile.dart
  so that abcompile.dart is not text scraping the numbers

* Adjust pkg/front_end/tool/fasta/compile.dart so that --verbose
  is not needed to capture the needed numbers and progress

* tweak the A/B summary section to paste more easily
  into Google Sheets

R=paulberry@google.com

Committed: https://github.com/dart-lang/sdk/commit/82685eacbff955f3b027e028c9b2450dd44720ac

[danrubel, 2017-03-03 16:56:10]

danrubel@google.com changed reviewers:
+ ahe@google.com, paulberry@google.com

[Paul Berry, 2017-03-03 18:08:53]

lgtm

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/ab...
File pkg/front_end/tool/fasta/abcompile.dart (right):

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/ab...
pkg/front_end/tool/fasta/abcompile.dart:82: print('Standard Deviation of the
Mean:');
Let's not confuse the issue by printing the technique for computing the
uncertainty.  Let's just say 'Uncertainty:'.

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/ab...
pkg/front_end/tool/fasta/abcompile.dart:96: print('  Standard Deviation of the
mean: $coldStdDevMean');
Similarly, let's say "Uncertainty:" here and on line 99.

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/st...
File pkg/front_end/tool/fasta/standard_deviation.dart (right):

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/st...
pkg/front_end/tool/fasta/standard_deviation.dart:12: double stdDev =
sqrt(numerator / elapsedTimes.length);
This should be:

`stdDev = sqrt(numerator / (elapsedTimes.length - 1));`.

The reason is complicated--see
https://en.wikipedia.org/wiki/Bessel%27s_correction for more information.

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/st...
pkg/front_end/tool/fasta/standard_deviation.dart:17: return stdDev /
sqrt(elapsedTimes.length);
This one is correct.

[danrubel, 2017-03-03 22:46:56]

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/ab...
File pkg/front_end/tool/fasta/abcompile.dart (right):

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/ab...
pkg/front_end/tool/fasta/abcompile.dart:82: print('Standard Deviation of the
Mean:');
On 2017/03/03 18:08:53, Paul Berry wrote:
> Let's not confuse the issue by printing the technique for computing the
> uncertainty.  Let's just say 'Uncertainty:'.

Done.

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/ab...
pkg/front_end/tool/fasta/abcompile.dart:96: print('  Standard Deviation of the
mean: $coldStdDevMean');
On 2017/03/03 18:08:53, Paul Berry wrote:
> Similarly, let's say "Uncertainty:" here and on line 99.

Done.

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/st...
File pkg/front_end/tool/fasta/standard_deviation.dart (right):

https://codereview.chromium.org/2731703003/diff/1/pkg/front_end/tool/fasta/st...
pkg/front_end/tool/fasta/standard_deviation.dart:12: double stdDev =
sqrt(numerator / elapsedTimes.length);
On 2017/03/03 18:08:53, Paul Berry wrote:
> This should be:
> 
> `stdDev = sqrt(numerator / (elapsedTimes.length - 1));`.
> 
> The reason is complicated--see
> https://en.wikipedia.org/wiki/Bessel%27s_correction for more information.

Done.

[Paul Berry, 2017-03-06 14:09:51]

lgtm

[danrubel, 2017-03-06 15:09:52]

Description was changed from

==========
A/B fasta performance measurement harness

Given two separate Dart SDK repo clones, A and B,
this pkg/front_end/tool/fasta/abcompile.dart harness
alternately runs

<A>/pkg/front_end/tool/fasta/compile.dart
then
<B>/pkg/front_end/tool/fasta/compile.dart

so that a performance comparison can be made of the difference
between fasta in A vs B.
For example, from the <A> root directory:

xcodebuild/ReleaseX64/dart
  -DbRoot=/path/to/2nd/dart/sdk/repo
  -Diterations=15
  pkg/front_end/tool/fasta/abcompile.dart
  --compile-sdk=xcodebuild/ReleaseX64/patched_sdk/
  pkg/compiler/lib/src/dart2js.dart
  --verbose

The output contains the raw data along with a summary
of the average difference, and is formatted in a way
that can be easily pasted into Google Sheets
for further analysis.

Future improvements to this include:

* Add JSON output to pkg/front_end/tool/fasta/compile.dart
  so that abcompile.dart is not text scraping the numbers

* Adjust pkg/front_end/tool/fasta/compile.dart so that --verbose
  is not needed to capture the needed numbers and progress

* tweak the A/B summary section to paste more easily
  into Google Sheets
==========

to

==========
A/B fasta performance measurement harness

Given two separate Dart SDK repo clones, A and B,
this pkg/front_end/tool/fasta/abcompile.dart harness
alternately runs

<A>/pkg/front_end/tool/fasta/compile.dart
then
<B>/pkg/front_end/tool/fasta/compile.dart

so that a performance comparison can be made of the difference
between fasta in A vs B.
For example, from the <A> root directory:

xcodebuild/ReleaseX64/dart
  -DbRoot=/path/to/2nd/dart/sdk/repo
  -Diterations=15
  pkg/front_end/tool/fasta/abcompile.dart
  --compile-sdk=xcodebuild/ReleaseX64/patched_sdk/
  pkg/compiler/lib/src/dart2js.dart
  --verbose

The output contains the raw data along with a summary
of the average difference, and is formatted in a way
that can be easily pasted into Google Sheets
for further analysis.

Future improvements to this include:

* Add JSON output to pkg/front_end/tool/fasta/compile.dart
  so that abcompile.dart is not text scraping the numbers

* Adjust pkg/front_end/tool/fasta/compile.dart so that --verbose
  is not needed to capture the needed numbers and progress

* tweak the A/B summary section to paste more easily
  into Google Sheets

R=paulberry@google.com

Committed:
https://github.com/dart-lang/sdk/commit/82685eacbff955f3b027e028c9b2450dd44720ac
==========

[danrubel, 2017-03-06 15:09:54]

Committed patchset #3 (id:40001) manually as
82685eacbff955f3b027e028c9b2450dd44720ac (presubmit successful).

[Vyacheslav Egorov (Google), 2017-03-06 15:32:52]

vegorov@google.com changed reviewers:
+ vegorov@google.com

[Vyacheslav Egorov (Google), 2017-03-06 15:32:53]

This commit has broken pkg/front_end/test/fasta/bootstrap_test test:

FAILED: none-vm-checked release_x64 pkg/front_end/test/fasta/bootstrap_test
Expected: Pass Slow 
Actual: RuntimeError
CommandOutput[vm]:

stdout:
unittest-suite-wait-for-done
out/ReleaseX64/patched_sdk/lib/_builtin/_builtin.dart:26:31: Error: Method not
found: 'main'.
Unhandled exception:
RangeError (start): Invalid value: Not in range 0..1, inclusive: 3
#0      RangeError.checkValidRange (dart:core/errors.dart:316)
#1      List.sublist (dart:core-patch/growable_array.dart:79)
#2      main
(file:///b/build/slave/vm-kernel-linux-release-x64-be/build/sdk/pkg/front_end/tool/fasta/compile.dart:30:40)
<asynchronous suspension>
#3      _startIsolate.<anonymous closure>
(dart:isolate-patch/isolate_patch.dart:259)
#4      _RawReceivePortImpl._handleMessage
(dart:isolate-patch/isolate_patch.dart:148)



stderr:
Unhandled exception:
Compilation failed.
#0      runCompiler
(file:///b/build/slave/vm-kernel-linux-release-x64-be/build/sdk/pkg/front_end/test/fasta/bootstrap_test.dart:65:5)
<asynchronous suspension>
#1      main
(file:///b/build/slave/vm-kernel-linux-release-x64-be/build/sdk/pkg/front_end/test/fasta/bootstrap_test.dart:34:11)
<asynchronous suspension>
#2      _startIsolate.<anonymous closure>
(dart:isolate-patch/isolate_patch.dart:261)
#3      _RawReceivePortImpl._handleMessage
(dart:isolate-patch/isolate_patch.dart:148)


Command[vm]: DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart --enable_asserts
--enable_type_checks --ignore-unrecognized-flags
--packages=/b/build/slave/vm-kernel-linux-release-x64-be/build/sdk/.packages
/b/build/slave/vm-kernel-linux-release-x64-be/build/sdk/pkg/front_end/test/fasta/bootstrap_test.dart
Took 0:00:10.352192

[danrubel, 2017-03-06 19:33:16]

On 2017/03/06 15:32:53, Vyacheslav Egorov (Google) wrote:
> This commit has broken pkg/front_end/test/fasta/bootstrap_test test

Fixed https://codereview.chromium.org/2734863003/