DDC: mostly incremental compilation, fixes #223

DDC: mostly incremental compilation, fixes #223
if the output file is out of date, we recompile, otherwise we skip as much work as possible

Further details:
Multitest support (Dart language tests) also was updated to avoid overwriting with identical data. For the same reason, codegen_test no longer deletes old output automatically.

If nothing changed, ./tool/build_sdk.dart and ./test/codengen_test.dart should be almost instantaneous.

If any of the compiler sources changed, a full rebuild is necessary. Right now it tries to detect this automatically, but it's a bit flaky because finding our own sources isn't reliable (e.g. test runner starts us using a data: URL, so Platform.script is useless). Maybe we should just try and have all output in one place, so you can rm -rf to clean it?

Definitely give this a look over ... there's always the possibility with this kind of change of introducing bugs due to stale output.

[Jennifer Messerly, 2015-09-03 20:34:04]

jmesserly@google.com changed reviewers:
+ leafp@google.com, vsm@google.com

[Jennifer Messerly, 2015-09-03 20:45:41]

btw, appreciate high level feedback on how we can structure this better. Feels
like _compileLibrary is more complicated than I'd like.

In an ideal world we'd just have another build system handle this, it's what
they do. However, crucially, we need to keep a warm AnalysisContext in memory,
until we get really really fast summaries. That seems to limit our options for
reusing other existing build tools. Best bet would be something in the NPM
ecosystem most likely. Let me know if you'd prefer that route. Also happy to
land this and keep iterating.

[Jennifer Messerly, 2015-09-03 20:46:24]

On 2015/09/03 20:45:41, John Messerly wrote:
> btw, appreciate high level feedback on how we can structure this better. Feels
> like _compileLibrary is more complicated than I'd like.
> 
> In an ideal world we'd just have another build system handle this, it's what
> they do. However, crucially, we need to keep a warm AnalysisContext in memory,
> until we get really really fast summaries. That seems to limit our options for
> reusing other existing build tools. Best bet would be something in the NPM
> ecosystem most likely. Let me know if you'd prefer that route. Also happy to
> land this and keep iterating.

oh, the other challenge with Dart is the dependencies require (some level of)
parsing source code, which also pulls in an in-memory analyzer.

[Leaf, 2015-09-04 21:46:31]

https://codereview.chromium.org/1322333003/diff/40001/lib/src/compiler.dart
File lib/src/compiler.dart (right):

https://codereview.chromium.org/1322333003/diff/40001/lib/src/compiler.dart#n...
lib/src/compiler.dart:138: if (!_compiled.add(source.uri)) return false;
I'm worried about this doing the right thing in the presence of cycles.  If I
have a cycle of libraries A and B, and I modify A, then I think this code does
the following:

1) Adds A to _compiled
2) Tries to compile B
3) Adds B to _compiled
4) Tries to compile A
5) Notices that A is in _compiled, so returns false for changed,
6) Does a time stamp check on B, sees that it hasn't itself changed, so doesn't
recompile it
7) Recompiles A based on timestamps

Am I missing something?

https://codereview.chromium.org/1322333003/diff/40001/lib/src/options.dart
File lib/src/options.dart (right):

https://codereview.chromium.org/1322333003/diff/40001/lib/src/options.dart#ne...
lib/src/options.dart:383: /// compiler itself changes.
At some point, we should probably include a version string in the generated code
so that we could invalidate even if we can't find the compiler sources.  Maybe a
TODO?

[Jennifer Messerly, 2015-09-09 16:15:40]

On 2015/09/04 21:46:31, Leaf wrote:
> https://codereview.chromium.org/1322333003/diff/40001/lib/src/compiler.dart
> File lib/src/compiler.dart (right):
> 
>
https://codereview.chromium.org/1322333003/diff/40001/lib/src/compiler.dart#n...
> lib/src/compiler.dart:138: if (!_compiled.add(source.uri)) return false;
> I'm worried about this doing the right thing in the presence of cycles.  If I
> have a cycle of libraries A and B, and I modify A, then I think this code does
> the following:
> 
> 1) Adds A to _compiled
> 2) Tries to compile B
> 3) Adds B to _compiled
> 4) Tries to compile A
> 5) Notices that A is in _compiled, so returns false for changed,
> 6) Does a time stamp check on B, sees that it hasn't itself changed, so
doesn't
> recompile it
> 7) Recompiles A based on timestamps
> 
> Am I missing something?
> 
> https://codereview.chromium.org/1322333003/diff/40001/lib/src/options.dart
> File lib/src/options.dart (right):
> 
>
https://codereview.chromium.org/1322333003/diff/40001/lib/src/options.dart#ne...
> lib/src/options.dart:383: /// compiler itself changes.
> At some point, we should probably include a version string in the generated
code
> so that we could invalidate even if we can't find the compiler sources.  Maybe
a
> TODO?

fyi, Leaf and I discussed this last week ... it's going to need some rework to
support library cycles.

Going to take a break from this for now, & come back to it when lang tests are
further along.