Separate compile-time and runtime errors.

Separate compile-time and runtime errors.

Committed: https://code.google.com/p/dart/source/detail?r=12964

[ahe, 2012-09-27 11:55:29]

This change to test.dart will allow dart2js to detect incorrect compile-time
errors or missing compile-time errors.

Here is an example that illustrates how these problems are diagnosed:

FAILED: dart2dart-vm release_ia32
co19/Language/10_Expressions/05_Strings/1_String_Interpolation_A01_t13
Expected: pass 
Actual: fail
Compile-time error expected.

stdout:
tests/co19/src/Language/10_Expressions/05_Strings/1_String_Interpolation_A01_t13.dart:24:5:
Warning: cannot resolve fooo
  "$fooo";
    ^^^^

Command line: xcodebuild/ReleaseIA32/dart2js --allow-mock-compilation
/.../dart/tests/co19/src/Language/10_Expressions/05_Strings/1_String_Interpolation_A01_t13.dart
--output-type=dart
--out=/.../dart/xcodebuild/ReleaseIA32/generated_tests/dart2dart-vm/tests_co19_src_Language_10_Expressions_05_Strings_1_String_Interpolation_A01_t13/out.dart

In this case, this is a bug in co19 test suite.

I'm still collecting status information for dart2dart.

[ahe, 2012-09-27 12:10:41]

Also, part of this change is needed to fix a bug with
dart/tools/testing/dart/co19_test.dart that caused me to get the status wrong
when testing dartc/analyzer on tests with runtime errors.

[Bill Hesse, 2012-09-27 12:55:38]

I think "incomplete" should be commented somewhere, that it means that not all
steps of the test case have been run.  Otherwise, you have to look at where this
value is created to know what it means.

Shouldn't we use a positive variant of the name, like "all_steps_run", or
"complete"?

LGTM.

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
File dart/tools/testing/dart/test_runner.dart (right):

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
dart/tools/testing/dart/test_runner.dart:249: // get anew TestOutput instance.
Typo: a new

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
File dart/tools/testing/dart/test_suite.dart (right):

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
dart/tools/testing/dart/test_suite.dart:364: if (info.hasRuntimeError &&
hasRuntime) {
Why not = info.hasCompileError ||
             hasRuntime && info.hasRuntimeError?

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
dart/tools/testing/dart/test_suite.dart:432: if (info.hasRuntimeError &&
hasRuntime) {
Why not ||?

[ahe, 2012-09-27 14:26:09]

On 2012/09/27 12:55:38, Bill Hesse wrote:
> I think "incomplete" should be commented somewhere, that it means that not all
> steps of the test case have been run.  Otherwise, you have to look at where
this
> value is created to know what it means.
> 
> Shouldn't we use a positive variant of the name, like "all_steps_run", or
> "complete"?

I have added documentation, but not renamed the variable. Testing test.dart is
really hard, and I would prefer not to have to do that again. I have some ideas
for how we can improve that situation by using something similar to
co19_test.dart.

> LGTM.
> 
>
http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
> File dart/tools/testing/dart/test_runner.dart (right):
> 
>
http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
> dart/tools/testing/dart/test_runner.dart:249: // get anew TestOutput instance.
> Typo: a new
> 
>
http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
> File dart/tools/testing/dart/test_suite.dart (right):
> 
>
http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
> dart/tools/testing/dart/test_suite.dart:364: if (info.hasRuntimeError &&
> hasRuntime) {
> Why not = info.hasCompileError ||
>              hasRuntime && info.hasRuntimeError?
> 
>
http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
> dart/tools/testing/dart/test_suite.dart:432: if (info.hasRuntimeError &&
> hasRuntime) {
> Why not ||?

[ahe, 2012-09-27 14:27:38]

Thank you, Bill!

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
File dart/tools/testing/dart/test_runner.dart (right):

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
dart/tools/testing/dart/test_runner.dart:249: // get anew TestOutput instance.
On 2012/09/27 12:55:38, Bill Hesse wrote:
> Typo: a new

Done.

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
File dart/tools/testing/dart/test_suite.dart (right):

http://codereview.chromium.org/10977054/diff/3001/dart/tools/testing/dart/tes...
dart/tools/testing/dart/test_suite.dart:364: if (info.hasRuntimeError &&
hasRuntime) {
On 2012/09/27 12:55:38, Bill Hesse wrote:
> Why not = info.hasCompileError ||
>              hasRuntime && info.hasRuntimeError?

I like to avoid expressions that span multiple lines.