Use status files for DDC's presubmit script/travis bot

Use status files for DDC's presubmit script/travis bot

This only handles compile time errors & compiler crashes. It does not yet support status files at runtime, that is still hard coded in language_tests.js. Some preliminary support for runtime status is there, but it's not enabled yet.

This also updates *_strong.status to reflect current test behavior. It looks like those files have not been triaged in a while.

R=rnystrom@google.com

Committed: https://github.com/dart-lang/sdk/commit/58a6227198d2aa927df19f1b4862f8d0017d6da4

[Jennifer Messerly, 2017-07-29 03:07:30]

Description was changed from

==========
Use existing status files for DDC's travis bot

Also update *_strong.status, as it looks like this has not been triaged in a
while.
This also contains some preliminary support for loading status information at
runtime, but it is not enabled yet.
==========

to

==========
Use status files for DDC's presubmit script/travis bot

This only handles compile time errors & compiler crashes. It does not yet
support status files at runtime, that is still hard coded in language_tests.js.
Some preliminary support for runtime status is there, but it's not enabled yet.

This also updates *_strong.status to reflect current test behavior. It looks
like those files have not been triaged in a while.
==========

[Jennifer Messerly, 2017-07-29 03:07:44]

jmesserly@google.com changed reviewers:
+ rnystrom@google.com, vsm@google.com

[Jennifer Messerly, 2017-07-29 03:08:33]

Also FYI: this builds on some of the other CL's I sent out tonight, so you may
want to check those smaller ones first.

[vsm, 2017-07-31 21:02:20]

https://codereview.chromium.org/2987093002/diff/10001/pkg/dev_compiler/test/c...
File pkg/dev_compiler/test/codegen_test.dart (right):

https://codereview.chromium.org/2987093002/diff/10001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:97: return; //!!!
Did you mean to leave that return here?

[Jennifer Messerly, 2017-07-31 21:24:55]

https://codereview.chromium.org/2987093002/diff/10001/pkg/dev_compiler/test/c...
File pkg/dev_compiler/test/codegen_test.dart (right):

https://codereview.chromium.org/2987093002/diff/10001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:97: return; //!!!
On 2017/07/31 21:02:20, vsm wrote:
> Did you mean to leave that return here?

Oops! after I got it passing with the compile error change, I started working on
`_writeRuntimeStatus` and debugging that a lot. I'll upload with that removed :)

[Jennifer Messerly, 2017-07-31 21:42:00]

PTAL

[Bob Nystrom, 2017-07-31 22:42:32]

LGTM!

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
File pkg/dev_compiler/test/codegen_test.dart (right):

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:40: import
'package:test_dart/options.dart';
I'm a little worried about importing code directly from test.dart because it
makes it easy for anyone hacking on test.dart to break DDC without realizing it.
But I guess it's the best approach for now. Certainly getting DDC's bots on the
main waterfall will help.

It would be good to email dart-testing@ and give them a heads up that they
should run DDC tests if they change test.dart.

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:282: // Skip passing tests.
"Skip" is confusing in this context. How about "Passing tests are the default,
so don't write them."

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:350: extractTestsFromMultitest(file,
contents, tests, {});
Remove the outcomes parameter since it's no longer being used?

https://codereview.chromium.org/2987093002/diff/30001/tests/language_strong/l...
File tests/language_strong/language_strong.status (right):

https://codereview.chromium.org/2987093002/diff/30001/tests/language_strong/l...
tests/language_strong/language_strong.status:128: await_test: CompileTimeError
Most of the above tests are gone now, aren't they? They should have been moved
to language_2.

[Jennifer Messerly, 2017-08-01 18:23:21]

thanks!

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
File pkg/dev_compiler/test/codegen_test.dart (right):

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:40: import
'package:test_dart/options.dart';
On 2017/07/31 22:42:32, Bob Nystrom wrote:
> I'm a little worried about importing code directly from test.dart because it
> makes it easy for anyone hacking on test.dart to break DDC without realizing
it.
> But I guess it's the best approach for now. Certainly getting DDC's bots on
the
> main waterfall will help.
> 
> It would be good to email dart-testing@ and give them a heads up that they
> should run DDC tests if they change test.dart.

Some mitigations:
* I hope to be able to delete this whole file shortly. This CL is a step along
that path.
* We could move the file to test_dart directory if that would help. Only DDC's
travis/presubmit would use it, tho, so IDK if that would help.
* It can't possibly be worse than the current setup :). Right now we're broken
by every status file or test change. Those have got to be more frequent than
test_dart changes :)

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:282: // Skip passing tests.
On 2017/07/31 22:42:32, Bob Nystrom wrote:
> "Skip" is confusing in this context. How about "Passing tests are the default,
> so don't write them."

Done.

https://codereview.chromium.org/2987093002/diff/30001/pkg/dev_compiler/test/c...
pkg/dev_compiler/test/codegen_test.dart:350: extractTestsFromMultitest(file,
contents, tests, {});
On 2017/07/31 22:42:32, Bob Nystrom wrote:
> Remove the outcomes parameter since it's no longer being used?

Done.

https://codereview.chromium.org/2987093002/diff/30001/tests/language_strong/l...
File tests/language_strong/language_strong.status (right):

https://codereview.chromium.org/2987093002/diff/30001/tests/language_strong/l...
tests/language_strong/language_strong.status:128: await_test: CompileTimeError
On 2017/07/31 22:42:32, Bob Nystrom wrote:
> Most of the above tests are gone now, aren't they? They should have been moved
> to language_2.

it's hard to tell ... every time I merge more stuff has changed :(

any advice on how to find entries that are gone in an easy way? if so that would
be really appreciated. otherwise I'm not sure an efficient way to find them.
(and the longer I wait to land this, the worse things get...)

[Bob Nystrom, 2017-08-01 20:48:39]

On 2017/08/01 18:23:21, Jennifer Messerly wrote:
> any advice on how to find entries that are gone in an easy way?

Not currently, no.

I spent a little time a while back hacking together some code that would scrape
a status file and print out any entries that don't match any files, but didn't
get around to finishing it. It had some false positives and, at least when I ran
it last week, didn't find too many true positives, so I put it on ice.

If you'd like to take a look, I could send you the code.

– bob

[Jennifer Messerly, 2017-08-01 23:32:19]

On 2017/08/01 20:48:39, Bob Nystrom wrote:
> On 2017/08/01 18:23:21, Jennifer Messerly wrote:
> > any advice on how to find entries that are gone in an easy way?
> 
> Not currently, no.
> 
> I spent a little time a while back hacking together some code that would
scrape
> a status file and print out any entries that don't match any files, but didn't
> get around to finishing it. It had some false positives and, at least when I
ran
> it last week, didn't find too many true positives, so I put it on ice.
> 
> If you'd like to take a look, I could send you the code.
> 
> – bob

Yes that'd be super helpful, if you've got it. If not, I have an idea on how I
can throw that together. I'll attempt to clean out dead entries in a followup.
And in the worst case, *_strong will go away, so the problem will fix itself :)

[Jennifer Messerly, 2017-08-01 23:36:26]

Description was changed from

==========
Use status files for DDC's presubmit script/travis bot

This only handles compile time errors & compiler crashes. It does not yet
support status files at runtime, that is still hard coded in language_tests.js.
Some preliminary support for runtime status is there, but it's not enabled yet.

This also updates *_strong.status to reflect current test behavior. It looks
like those files have not been triaged in a while.
==========

to

==========
Use status files for DDC's presubmit script/travis bot

This only handles compile time errors & compiler crashes. It does not yet
support status files at runtime, that is still hard coded in language_tests.js.
Some preliminary support for runtime status is there, but it's not enabled yet.

This also updates *_strong.status to reflect current test behavior. It looks
like those files have not been triaged in a while.

R=rnystrom@google.com

Committed:
https://github.com/dart-lang/sdk/commit/58a6227198d2aa927df19f1b4862f8d0017d6da4
==========

[Jennifer Messerly, 2017-08-01 23:36:28]

Committed patchset #4 (id:50001) manually as
58a6227198d2aa927df19f1b4862f8d0017d6da4 (presubmit successful).