Add a simple benchmark for analysis and error reporting.

Add a simple benchmark for analysis and error reporting.

R=brianwilkerson@google.com

Committed: https://github.com/dart-lang/sdk/commit/d09b21f63d625f28e04230befa9ca328ef5cdf66

[Bob Nystrom, 2016-03-22 18:24:38]

rnystrom@google.com changed reviewers:
+ brianwilkerson@google.com

[Bob Nystrom, 2016-03-22 18:24:38]

This benchmark is a little dubious because it uses analyzer's own codebase as
its dataset. This means changes to analyzer may affect the output, so it doesn't
maintain a great long-term baseline.

But, it's enough to be roughly useful for profiling and trying out changes to
analyzer, and I wanted to start getting some kind of benchmarking going.

What do you think?

[Brian Wilkerson, 2016-03-22 20:07:54]

We used to have a similar benchmark that used an infrequently updated copy of
dart2js, but that code, which was behind a hidden flag in dartanalyzer was
removed because it was no longer being run. If we really care about the change
to the code being analyzed, I suppose we could pin the test to a specific
version of some GitHub repo, but I doubt it's necessary.

This benchmark looks good. It mimics fairly closely, but not exactly, the
process that the command-line analyzer uses (which differs from what server
uses).

We might want to just re-implement the benchmark directly in dartanalyzer. That
approach had the advantage that we were measuring exactly what user was running,
so as the code in analyzer was changed the benchmark would change. As it is in
this CL we skip a lot of work that analyzer has to do.

Either way, though, given that this is more if a benchmark of how dartanalyzer
works we might want to move it to the analyzer_cli repo.

But I don't want to hold anyone up needlessly, so you can commit it as is if you
want. LGTM.

[Bob Nystrom, 2016-03-23 00:34:23]

On 2016/03/22 20:07:54, Brian Wilkerson wrote:
> We used to have a similar benchmark that used an infrequently updated copy of
> dart2js, but that code, which was behind a hidden flag in dartanalyzer was
> removed because it was no longer being run.

Yeah, I thought about doing something like that, but it seemed needlessly fussy.

I think it's pretty unlikely that optimization would itself make such sweeping
changes to analyzer's code to affect its use as input for the benchmark.

> This benchmark looks good. It mimics fairly closely, but not exactly, the
> process that the command-line analyzer uses (which differs from what server
> uses).

What's different about how the server works?

> We might want to just re-implement the benchmark directly in dartanalyzer.

I'd like to keep this (and eventually other) benchmarks separate from the
application. I look at this like a test. It's job is to validate some part of
the program, but not ship with it.

Instrumenting the actual application is also useful, but I think of that as a
different thing from a benchmark.

> As it is in this CL we skip a lot of work that analyzer has to do.

I wasn't necessarily going for a full end-to-end benchmark here, but I'm
definitely interested to know what it's missing. Is there anything else I should
add in?

> Either way, though, given that this is more if a benchmark of how dartanalyzer
> works we might want to move it to the analyzer_cli repo.

All of the code it exercises is in analyzer itself, so that seems weird to me.
My goal was more to benchmark what analyzer does, not necessarily simulate how
the command-line analyzer works.

If you want, I'm definitely up for making changes to the benchmark. It was the
simplest thing I could get work that, hopefully, covered most of what a "full"
analysis does.

> But I don't want to hold anyone up needlessly, so you can commit it as is if
you
> want. LGTM.

Thanks! Launch and iterate sounds good to me. I just want to start having some
kind of suite of benchmarks in place so we can do repeatable, measurable
performance work.

– bob

[Bob Nystrom, 2016-03-23 00:35:22]

Description was changed from

==========
Add a simple benchmark for analysis and error reporting.
==========

to

==========
Add a simple benchmark for analysis and error reporting.

R=brianwilkerson@google.com

Committed:
https://github.com/dart-lang/sdk/commit/d09b21f63d625f28e04230befa9ca328ef5cdf66
==========

[Bob Nystrom, 2016-03-23 00:35:23]

Committed patchset #1 (id:1) manually as
d09b21f63d625f28e04230befa9ca328ef5cdf66 (presubmit successful).