Redo our benchmarks; add a benchmark/benchmarks.dart tool.

Redo our benchmarks; add a benchmark/benchmarks.dart tool.

BUG=
R=brianwilkerson@google.com, scheglov@google.com

Committed: https://github.com/dart-lang/sdk/commit/f383c17bc288bdcc049010f98e968ba6f5e5dd79

[devoncarew, 2017-08-01 23:09:59]

devoncarew@google.com changed reviewers:
+ brianwilkerson@google.com, scheglov@google.com

[devoncarew, 2017-08-01 23:10:00]

Add 6 new benchmarks for the analysis server:
- these benchmarks are available by running the tool benchmark/benchmarks.dart
list
- they can be run via benchmark/benchmarks.dart run <benchmark-id>
- each benchmark produces a few performance metrics
- these are intended to be driven in an automated way from our performance CI
- new benchmarks will be easy to add, and are easy to run locally
- the benchmarks are tested via unit tests, so we'll know when they regress
(from a stability POV, not a performance one)
- we're currently benchmarking against the pkg/analysis_server codebase (and
making edits and completions against lib/src/analysis_server.dart)

Current generated metrics are:
- analysis-server-cold-analysis
- analysis-server-cold-memory
- analysis-server-warm-analysis
- analysis-server-warm-memory
- analysis-server-edit
- analysis-server-completion

Comments welcome! I would like to:
- land and iterate these
- switch the performance CI to using them
- and then remove the older benchmarks

[scheglov, 2017-08-02 00:12:00]

lgtm

[Brian Wilkerson, 2017-08-02 13:52:48]

lgtm

https://codereview.chromium.org/2986293002/diff/1/pkg/analysis_server/test/be...
File pkg/analysis_server/test/benchmarks_test.dart (right):

https://codereview.chromium.org/2986293002/diff/1/pkg/analysis_server/test/be...
pkg/analysis_server/test/benchmarks_test.dart:1: // Copyright (c) 2014, the Dart
project authors.  Please see the AUTHORS file
2017

[devoncarew, 2017-08-02 15:04:48]

https://codereview.chromium.org/2986293002/diff/1/pkg/analysis_server/test/be...
File pkg/analysis_server/test/benchmarks_test.dart (right):

https://codereview.chromium.org/2986293002/diff/1/pkg/analysis_server/test/be...
pkg/analysis_server/test/benchmarks_test.dart:1: // Copyright (c) 2014, the Dart
project authors.  Please see the AUTHORS file
On 2017/08/02 13:52:47, Brian Wilkerson wrote:
> 2017

Done.

[devoncarew, 2017-08-02 15:07:00]

The three benchmark tests currently take 7-10 seconds to run in total; so
slower, but on the order of our integration tests. I think this is worth it
given that our benchmarks will now be tested.

[devoncarew, 2017-08-02 15:07:39]

Description was changed from

==========
Redo our benchmarks; add a benchmark/benchmarks.dart tool.

BUG=
==========

to

==========
Redo our benchmarks; add a benchmark/benchmarks.dart tool.

BUG=
R=brianwilkerson@google.com, scheglov@google.com

Committed:
https://github.com/dart-lang/sdk/commit/f383c17bc288bdcc049010f98e968ba6f5e5dd79
==========

[devoncarew, 2017-08-02 15:07:40]

Committed patchset #2 (id:20001) manually as
f383c17bc288bdcc049010f98e968ba6f5e5dd79 (presubmit successful).

[sortie, 2017-08-03 21:31:51]

sortie@google.com changed reviewers:
+ sortie@google.com

[sortie, 2017-08-03 21:31:52]

The readme incorrectly mentions a benchmarks/ directory rather than a benchmark/
directory. It's unclear to me whether the dart sdk as a whole would benefit from
a unified system for testing benchmarks from any team. I need to think a bit
more in depth about which benchmarking responsibilities could benefit from being
offloaded onto the repository, and which shouldn't, and how.

[devoncarew, 2017-08-03 22:09:43]

On 2017/08/03 21:31:52, sortie wrote:
> The readme incorrectly mentions a benchmarks/ directory rather than a
benchmark/
> directory. It's unclear to me whether the dart sdk as a whole would benefit
from
> a unified system for testing benchmarks from any team. I need to think a bit
> more in depth about which benchmarking responsibilities could benefit from
being
> offloaded onto the repository, and which shouldn't, and how.

Thanks! Will update the readme in a follow up to be more correct. The general
motivation here is to:
- make the benchmarks easy for team members to run (and easy to debug new ones)
- make sure that the benchmarks can run commit-to-commit (that they don't
regress in terms of their ability to execute)
- make it easy to add new benchmarks (a more dynamic discovery system)

It sounds like the last feature may not really be achievable with the current
system.

[sortie, 2017-08-03 22:16:30]

Well, the contributor overhead can be reduced to a CL in the dart sdk adding the
benchmark, and another trivial CL to add the benchmark's name, what it outputs,
and when it was introduced to the list of which benchmarks exist. I think this
is manageable enough and can be documented. The second CL will also help sanity
check things on my end.