Add test that test.dart returns non-zero exit code for failing runs.

Add test that test.dart returns non-zero exit code for failing runs.
Additionally test that the buildbot output contains the
@@@STEP_FAILURE@@@ marker.

Rico and Emily, any other signals than exit code and @@@STEP_FAILURE@@@
that the buildbot uses to make builds red? I want to attempt to get
this tested so it will be much harder to accidentally make failing
builds turn up green.

R=ahe@google.com,ricow@google.com,efortuna@google.com
BUG=

Committed: https://code.google.com/p/dart/source/detail?r=13343

[ricow1, 2012-10-06 16:38:52]

LGTM, thanks a lot, this is great.
There will still be cases where we can mess things, but this is a very nice
sanity check.
Regarding the buildbot, it is rather stupid in respect to redness:
An annotated step is red iff:
@@@STEP_FAILURE@@@ 
is there
The 'master' annotated step will go red if the exitcode is not 0, otherwise it
will be green.

(the above is one of the reasons I don't like annotated steps based builders -
it seems fragile. But I am fully aware of the added benefits as well)

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_script.dart (right):

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_script.dart:5: // Simulates the a use
of test_progress during a failing run of test.dart.
the a -> the

[ricow1, 2012-10-06 16:40:38]

On 2012/10/06 16:38:52, ricow1 wrote:
> LGTM, thanks a lot, this is great.
> There will still be cases where we can mess things, but this is a very nice
> sanity check.
> Regarding the buildbot, it is rather stupid in respect to redness:
> An annotated step is red iff:
> @@@STEP_FAILURE@@@ 
> is there
> The 'master' annotated step will go red if the exitcode is not 0, otherwise it
> will be green.
> 
> (the above is one of the reasons I don't like annotated steps based builders -
> it seems fragile. But I am fully aware of the added benefits as well)
> 
>
https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
> File tests/standalone/io/test_runner_exit_code_script.dart (right):
> 
>
https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
> tests/standalone/io/test_runner_exit_code_script.dart:5: // Simulates the a
use
> of test_progress during a failing run of test.dart.
> the a -> the

For reference:
http://www.chromium.org/developers/testing/chromium-build-infrastructure/buil...

[ahe, 2012-10-08 07:12:37]

LGTM

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_test.dart (right):

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_test.dart:9: Process.run(executable,
[script, iterator.next()]).then((result) {
You should compare the string returned from iterator.next() to 'buildbot'. If
they are equal, the result should contain STEP_FAILURE.

[Mads Ager (google), 2012-10-08 07:28:45]

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_script.dart (right):

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_script.dart:5: // Simulates the a use
of test_progress during a failing run of test.dart.
On 2012/10/06 16:38:52, ricow1 wrote:
> the a -> the

Done.

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_test.dart (right):

https://codereview.chromium.org/11078007/diff/1/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_test.dart:9: Process.run(executable,
[script, iterator.next()]).then((result) {
On 2012/10/08 07:12:37, ahe wrote:
> You should compare the string returned from iterator.next() to 'buildbot'. If
> they are equal, the result should contain STEP_FAILURE.

Done.

[ahe, 2012-10-08 07:58:55]

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_test.dart (right):

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_test.dart:7: void runTests(String
executable, String script, Iterator iterator) {
Consider using a list instead of an iterator...

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_test.dart:9: var progressIndicator =
iterator.next();
... then these two lines can be:

for (String progressIndicator in list) {

[Mads Ager (google), 2012-10-08 08:00:49]

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_test.dart (right):

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_test.dart:9: var progressIndicator =
iterator.next();
On 2012/10/08 07:58:56, ahe wrote:
> ... then these two lines can be:
> 
> for (String progressIndicator in list) {

They could, but then you would start all the processes at once. I'm doing it
this way so we start one process at a time. I use the iterator to hold the state
between callbacks through the event loop.

[ahe, 2012-10-08 08:02:42]

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_test.dart (right):

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_test.dart:9: var progressIndicator =
iterator.next();
On 2012/10/08 08:00:49, Mads Ager wrote:
> On 2012/10/08 07:58:56, ahe wrote:
> > ... then these two lines can be:
> > 
> > for (String progressIndicator in list) {
> 
> They could, but then you would start all the processes at once. I'm doing it
> this way so we start one process at a time. I use the iterator to hold the
state
> between callbacks through the event loop.

Doh. I misread the code :-)

[Emily Fortuna, 2012-10-08 17:50:06]

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
File tests/standalone/io/test_runner_exit_code_test.dart (right):

https://codereview.chromium.org/11078007/diff/6/tests/standalone/io/test_runn...
tests/standalone/io/test_runner_exit_code_test.dart:13:
Expect.isTrue(result.stdout.contains("@@@STEP_FAILURE@@@"));
The "@@@STEP_FAILURE@@" string is only printed out for dart2js and browser bots.
The dart2dart, dartc and dart VM bots only use the exit code instead to detect
failure.