change presubmit to run co19 tests

Change presubmit to run co19 tests, but run a few less frog configurations to keep the total runtime similar.

Committed: https://code.google.com/p/dart/source/detail?r=1471

[Jennifer Messerly, 2011-11-10 23:40:06]

Hi guys,

I'd love to hear comments on this change. I'm trying to make presubmit.py catch
a higher percentage of things that might fail on the build bots.

+co19 tests on frogsh
-all tests on frog, except corelib (and of course the selfhost build itself)
-some of the selfhost permutations in the beginning. I was partly responsible
for this bloat, apologies :)

Total runtimes are very similar-see below. It's slightly shorter in wall-clock
time, slightly longer in total compute time.

Cheers,
- John


****** Before ****** 
Started
Compiling on Dart VM took 12.7324988842 seconds in checked mode + compile all
Compiling on Dart VM took 12.8261489868 seconds in checked mode
Compiling on Dart VM took 3.34634900093 seconds
Bootstrapping took 2.89441990852 seconds in checked mode
Generated checked frogsh is 694 kB
Total: 2022 tests
 *   21 tests will be skipped
 *    0 tests are expected to be flaky but not crash
 * 1547 tests are expected to pass
 *    0 tests are expected to fail that we won't fix
 *  454 tests are expected to fail that we should fix
 *    0 tests are expected to crash that we should fix
 *    0 tests are running in batch mode
[01:31|% 100|+ 2001|-   0]: Done                                              

real	2m14.286s
user	28m19.245s
sys	2m52.978s


****** After ****** 
Started
Compiling on Dart VM took 12.385076046 seconds in checked mode + compile all
Bootstrapping took 2.90594506264 seconds in checked mode
Generated checked frogsh is 694 kB
Total: 63 tests
 *    4 tests will be skipped
 *    0 tests are expected to be flaky but not crash
 *   35 tests are expected to pass
 *    0 tests are expected to fail that we won't fix
 *   24 tests are expected to fail that we should fix
 *    0 tests are expected to crash that we should fix
 *    0 tests are running in batch mode
[00:03|% 100|+  59|-   0]: Done                                               
Total: 1348 tests
 *   14 tests will be skipped
 *    0 tests are expected to be flaky but not crash
 * 1031 tests are expected to pass
 *    0 tests are expected to fail that we won't fix
 *  303 tests are expected to fail that we should fix
 *    0 tests are expected to crash that we should fix
 *    0 tests are running in batch mode
[00:57|% 100|+ 1334|-   0]: Done                                              
Total: 1477 tests
 *   31 tests will be skipped
 *    0 tests are expected to be flaky but not crash
 *  958 tests are expected to pass
 *    0 tests are expected to fail that we won't fix
 *  488 tests are expected to fail that we should fix
 *    0 tests are expected to crash that we should fix
 *    0 tests are running in batch mode
[00:38|% 100|+ 1446|-   0]: Done                                              

real	2m3.301s
user	28m44.663s
sys	3m8.301s

http://codereview.chromium.org/8528015/diff/1/frog/presubmit.py
File frog/presubmit.py (right):

http://codereview.chromium.org/8528015/diff/1/frog/presubmit.py#newcode59
frog/presubmit.py:59: '--compile_all', '--enable_type_checks', '--out=frogsh',
--compile-all was being passed to the VM instead of frog, so wasn't having its
intended effect.

http://codereview.chromium.org/8528015/diff/1/frog/presubmit.py#newcode75
frog/presubmit.py:75: test_cmd = ['../tools/test.py', '--report',
'--timeout=10',
one of the co19 tests is testing sort and sometimes hits the 5s limit on my
machine.

[Siggi Cherem (dart-lang), 2011-11-11 00:10:53]

lgtm

this sounds good to me.

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py
File frog/presubmit.py (right):

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py#newcode75
frog/presubmit.py:75: test_cmd = ['../tools/test.py', '--report',
'--timeout=10',
why the increase to 10?

[Jennifer Messerly, 2011-11-11 00:21:13]

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py
File frog/presubmit.py (right):

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py#newcode75
frog/presubmit.py:75: test_cmd = ['../tools/test.py', '--report',
'--timeout=10',
On 2011/11/11 00:10:53, sigmund wrote:
> why the increase to 10? 

Oops, I think my comment got lost when I uploaded a new version. See here:
http://codereview.chromium.org/8528015/diff/1/frog/presubmit.py#newcode75

[Siggi Cherem (dart-lang), 2011-11-11 00:33:23]

got it - thanks! :)

[ahe, 2011-11-11 12:46:11]

LGTM!

http://codereview.chromium.org/8528015/diff/1/frog/presubmit.py
File frog/presubmit.py (right):

http://codereview.chromium.org/8528015/diff/1/frog/presubmit.py#newcode59
frog/presubmit.py:59: '--compile_all', '--enable_type_checks', '--out=frogsh',
On 2011/11/10 23:40:06, John Messerly wrote:
> --compile-all was being passed to the VM instead of frog, so wasn't having its
> intended effect.

That was the intention when I put it in :-)

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py
File frog/presubmit.py (right):

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py#newcode58
frog/presubmit.py:58: '--vm_flags=--enable_type_checks --enable_asserts', '--',
Could we keep --compile_all for dart_bin?

http://codereview.chromium.org/8528015/diff/3001/tests/co19/co19-frog.status
File tests/co19/co19-frog.status (right):

http://codereview.chromium.org/8528015/diff/3001/tests/co19/co19-frog.status#...
tests/co19/co19-frog.status:498: LibTest/core/Math/asin/Math/asin/A01/t01: Skip
# different results on Mac
Try: "Fail, Pass" instead of "Skip".

[jimhug, 2011-11-11 14:45:39]

LGTM!

However, I'm thinking I might need to upgrade to a desktop machine - the
starting numbers on my laptop are roughly 5x worse than you see...

I'd like to claim this policy for presubmit.py.  The purpose of this script is
to catch as many errors as possible in <3 minutes on a high-end desktop machine.
 As our test suite grows, we will keep updating this script to best meet those
goals.  If we ever hit the point where we believe that more testing than this is
required for typical checkins, then we will investigate other options such as
the gcl try/git-try.

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py
File frog/presubmit.py (right):

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py#newcode58
frog/presubmit.py:58: '--vm_flags=--enable_type_checks --enable_asserts', '--',
On 2011/11/11 12:46:11, ahe wrote:
> Could we keep --compile_all for dart_bin?

I agree that we should keep this to maximize coverage in the VM.  It's weird to
realize that Peter added this flag only two weeks ago and at the time we had no
idea how long it would be until this would also work for frog itself...

[Jennifer Messerly, 2011-11-11 18:10:02]

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py
File frog/presubmit.py (right):

http://codereview.chromium.org/8528015/diff/3001/frog/presubmit.py#newcode58
frog/presubmit.py:58: '--vm_flags=--enable_type_checks --enable_asserts', '--',
On 2011/11/11 14:45:39, jimhug wrote:
> On 2011/11/11 12:46:11, ahe wrote:
> > Could we keep --compile_all for dart_bin?
> 
> I agree that we should keep this to maximize coverage in the VM.  It's weird
to
> realize that Peter added this flag only two weeks ago and at the time we had
no
> idea how long it would be until this would also work for frog itself...

Haha, yeah, I didn't know this was meant to test the VM! I'll fix.