Verify status of statically failing tests

Verify status of statically failing tests

This should catch tests that suddenly start failing (or passing).

This is an admittedly contrived approach to track compilation failures in
the same runtime logic: the statically failing tests are now compiled
to a trivial output that throws a particular exception.  Ideally, we
wouldn't run these, but it's beneficial to keep status in the same
place.

On my laptop, this only adds ~8s to presubmit (which takes ~2:30 min).

Ideas on alternative approaches welcome.  :-)

R=jmesserly@google.com, leafp@google.com

Committed: https://github.com/dart-lang/dev_compiler/commit/2c19b4520cb748f52e01e44c70a588a383c68b97

[vsm, 2016-06-20 16:51:08]

Description was changed from

==========
Verify status of statically failing tests

This should catch tests that suddenly start failing (or passing).

This is a somewhat contrived approach to track compilation failures in
the same runtime logic: the statically failing tests are now compiled
to a trivial output that throws a particular exception.  Ideally, we
wouldn't run these, but it's beneficial to keep status in the same
place.

On my laptop, this only adds ~8s to presubmit (which takes ~2:30 min).
==========

to

==========
Verify status of statically failing tests

This should catch tests that suddenly start failing (or passing).

This is an admittedly contrived approach to track compilation failures in
the same runtime logic: the statically failing tests are now compiled
to a trivial output that throws a particular exception.  Ideally, we
wouldn't run these, but it's beneficial to keep status in the same
place.

On my laptop, this only adds ~8s to presubmit (which takes ~2:30 min).

Ideas on alternative approaches welcome.  :-)
==========

[vsm, 2016-06-20 16:51:08]

vsm@google.com changed reviewers:
+ jmesserly@google.com, leafp@google.com

[Leaf, 2016-06-21 17:11:16]

lgtm

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests.js
File test/browser/language_tests.js (right):

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests...
test/browser/language_tests.js:16: //   'deprecated' - test is not strong mode
safe
I'm worried this is a little misleading.  There are two kinds of non-strong mode
safe tests: those which are testing non-strong mode behavior (these are in fact
deprecated) and those which are testing behavior that should work in strong
mode, but which have incidental strong mode errors (more like un-triaged or
unfixed).  I wonder if there should be 'deprecated' for the former, and
something like 'untriaged' or 'notyetstrong' for the second?

[Jennifer Messerly, 2016-06-21 17:18:54]

lgtm

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests.js
File test/browser/language_tests.js (right):

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests...
test/browser/language_tests.js:16: //   'deprecated' - test is not strong mode
safe
On 2016/06/21 17:11:16, Leaf wrote:
> I'm worried this is a little misleading.  There are two kinds of non-strong
mode
> safe tests: those which are testing non-strong mode behavior (these are in
fact
> deprecated) and those which are testing behavior that should work in strong
> mode, but which have incidental strong mode errors (more like un-triaged or
> unfixed).  I wonder if there should be 'deprecated' for the former, and
> something like 'untriaged' or 'notyetstrong' for the second?

+1, I like both of those name suggestions!

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests...
test/browser/language_tests.js:2838: throw new Error(
tabs here?

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests...
test/browser/language_tests.js:2843: throw new Error(
here too

[vsm, 2016-06-22 13:26:01]

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests.js
File test/browser/language_tests.js (right):

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests...
test/browser/language_tests.js:16: //   'deprecated' - test is not strong mode
safe
On 2016/06/21 17:18:54, John Messerly wrote:
> On 2016/06/21 17:11:16, Leaf wrote:
> > I'm worried this is a little misleading.  There are two kinds of non-strong
> mode
> > safe tests: those which are testing non-strong mode behavior (these are in
> fact
> > deprecated) and those which are testing behavior that should work in strong
> > mode, but which have incidental strong mode errors (more like un-triaged or
> > unfixed).  I wonder if there should be 'deprecated' for the former, and
> > something like 'untriaged' or 'notyetstrong' for the second?
> 
> +1, I like both of those name suggestions!

Went with 'notyetstrong'.

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests...
test/browser/language_tests.js:2838: throw new Error(
On 2016/06/21 17:18:54, John Messerly wrote:
> tabs here?

doh!

https://codereview.chromium.org/2079373002/diff/1/test/browser/language_tests...
test/browser/language_tests.js:2843: throw new Error(
On 2016/06/21 17:18:54, John Messerly wrote:
> here too

Done.

[vsm, 2016-06-22 13:26:12]

Description was changed from

==========
Verify status of statically failing tests

This should catch tests that suddenly start failing (or passing).

This is an admittedly contrived approach to track compilation failures in
the same runtime logic: the statically failing tests are now compiled
to a trivial output that throws a particular exception.  Ideally, we
wouldn't run these, but it's beneficial to keep status in the same
place.

On my laptop, this only adds ~8s to presubmit (which takes ~2:30 min).

Ideas on alternative approaches welcome.  :-)
==========

to

==========
Verify status of statically failing tests

This should catch tests that suddenly start failing (or passing).

This is an admittedly contrived approach to track compilation failures in
the same runtime logic: the statically failing tests are now compiled
to a trivial output that throws a particular exception.  Ideally, we
wouldn't run these, but it's beneficial to keep status in the same
place.

On my laptop, this only adds ~8s to presubmit (which takes ~2:30 min).

Ideas on alternative approaches welcome.  :-)

R=jmesserly@google.com, leafp@google.com

Committed:
https://github.com/dart-lang/dev_compiler/commit/2c19b4520cb748f52e01e44c70a5...
==========

[vsm, 2016-06-22 13:26:14]

Committed patchset #2 (id:20001) manually as
2c19b4520cb748f52e01e44c70a588a383c68b97 (presubmit successful).