Measure time spent building graph of files.

Measure time spent building graph of files.

It's about 1000 ms for analyzer_cli, which I use for benchmarking.

R=ahe@google.com, paulberry@google.com, sigmund@google.com
BUG=

Committed: https://github.com/dart-lang/sdk/commit/710af5af38319bd233ec211f50a375c5d37d59bd

[Paul Berry, 2017-06-07 17:14:22]

lgtm

[scheglov, 2017-06-07 17:26:14]

Description was changed from

==========
Measure time spent building graph of files.

It's about 1000 ms for analyzer_cli, which I use for benchmarking.

R=ahe@google.com, paulberry@google.com, sigmund@google.com
BUG=
==========

to

==========
Measure time spent building graph of files.

It's about 1000 ms for analyzer_cli, which I use for benchmarking.

R=ahe@google.com, paulberry@google.com, sigmund@google.com
BUG=

Committed:
https://github.com/dart-lang/sdk/commit/710af5af38319bd233ec211f50a375c5d37d59bd
==========

[scheglov, 2017-06-07 17:26:15]

Committed patchset #1 (id:1) manually as
710af5af38319bd233ec211f50a375c5d37d59bd (presubmit successful).

[ahe, 2017-06-07 17:30:39]

DBC: Fasta builds an unlinked outline of analyzer_cli in about 2000ms, so it is
possible that this could be faster.

Besides building a graph of files, do you collect any other data?

Am I understanding correctly that this is about finding library directives in
all files and creating the graph they imply?

[scheglov, 2017-06-07 17:35:05]

On 2017/06/07 17:30:39, ahe wrote:
> DBC: Fasta builds an unlinked outline of analyzer_cli in about 2000ms, so it
is
> possible that this could be faster.
> 
> Besides building a graph of files, do you collect any other data?
> 
> Am I understanding correctly that this is about finding library directives in
> all files and creating the graph they imply?

We read files, compute content hashes, parse directives and walk the references.
I agree that 1000 ms is bigger than I'd like and plan to look how to make in
faster.
But we cannot optimize what we don't measure, so the CL :-)

[ahe, 2017-06-07 17:41:24]

On 2017/06/07 17:35:05, scheglov wrote:
> On 2017/06/07 17:30:39, ahe wrote:
> > DBC: Fasta builds an unlinked outline of analyzer_cli in about 2000ms, so it
> is
> > possible that this could be faster.
> > 
> > Besides building a graph of files, do you collect any other data?
> > 
> > Am I understanding correctly that this is about finding library directives
in
> > all files and creating the graph they imply?
> 
> We read files, compute content hashes, parse directives and walk the
references.
> I agree that 1000 ms is bigger than I'd like and plan to look how to make in
> faster.
> But we cannot optimize what we don't measure, so the CL :-)

Yes, obviously. I didn't mean to imply that there were any problems with this
CL.

I didn't think about content hashes, those might be costly.

Anyway, I don't mean to imply you're doing anything wrong, I'm mostly concerned
that we don't have the right API for the scanner to help with the directives but
I'm also getting some ideas that I look forward to discussing with Dan :-)

[scheglov, 2017-06-07 17:45:21]

On 2017/06/07 17:41:24, ahe wrote:
> On 2017/06/07 17:35:05, scheglov wrote:
> > On 2017/06/07 17:30:39, ahe wrote:
> > > DBC: Fasta builds an unlinked outline of analyzer_cli in about 2000ms, so
it
> > is
> > > possible that this could be faster.
> > > 
> > > Besides building a graph of files, do you collect any other data?
> > > 
> > > Am I understanding correctly that this is about finding library directives
> in
> > > all files and creating the graph they imply?
> > 
> > We read files, compute content hashes, parse directives and walk the
> references.
> > I agree that 1000 ms is bigger than I'd like and plan to look how to make in
> > faster.
> > But we cannot optimize what we don't measure, so the CL :-)
> 
> Yes, obviously. I didn't mean to imply that there were any problems with this
> CL.
> 
> I didn't think about content hashes, those might be costly.
> 
> Anyway, I don't mean to imply you're doing anything wrong, I'm mostly
concerned
> that we don't have the right API for the scanner to help with the directives
but
> I'm also getting some ideas that I look forward to discussing with Dan :-)

For directives I'm going to employ a technique that does not require any
scanning and parsing in the most cases. We can cache this information with the
content hash as a key. Each file changes rarely, so in the most cases we will
able to read directive URIs from the cache. Initial / cold compilation will pay
additional scanning and parsing, but I think that in practical cases, outside of
benchmarks, it will not matter.

[ahe, 2017-06-07 17:52:17]

On 2017/06/07 17:45:21, scheglov wrote:
> On 2017/06/07 17:41:24, ahe wrote:
> > On 2017/06/07 17:35:05, scheglov wrote:
> > > On 2017/06/07 17:30:39, ahe wrote:
> > > > DBC: Fasta builds an unlinked outline of analyzer_cli in about 2000ms,
so
> it
> > > is
> > > > possible that this could be faster.
> > > > 
> > > > Besides building a graph of files, do you collect any other data?
> > > > 
> > > > Am I understanding correctly that this is about finding library
directives
> > in
> > > > all files and creating the graph they imply?
> > > 
> > > We read files, compute content hashes, parse directives and walk the
> > references.
> > > I agree that 1000 ms is bigger than I'd like and plan to look how to make
in
> > > faster.
> > > But we cannot optimize what we don't measure, so the CL :-)
> > 
> > Yes, obviously. I didn't mean to imply that there were any problems with
this
> > CL.
> > 
> > I didn't think about content hashes, those might be costly.
> > 
> > Anyway, I don't mean to imply you're doing anything wrong, I'm mostly
> concerned
> > that we don't have the right API for the scanner to help with the directives
> but
> > I'm also getting some ideas that I look forward to discussing with Dan :-)
> 
> For directives I'm going to employ a technique that does not require any
> scanning and parsing in the most cases. We can cache this information with the
> content hash as a key. Each file changes rarely, so in the most cases we will
> able to read directive URIs from the cache. Initial / cold compilation will
pay
> additional scanning and parsing, but I think that in practical cases, outside
of
> benchmarks, it will not matter.

Good point.