Add the ability to run test suites in parallel.

Add the ability to run test suites in parallel.

R=kevmoo@google.com

Committed: https://github.com/dart-lang/test/commit/f0d5c2fad78de8346d9b33162f07f6006c3661da

[kevmoo, 2015-04-06 22:28:47]

On 2015/04/06 21:56:44, nweiz wrote:

Code change LGTM, but I get 4 errors running locally (with bleeding edge SDK
build)

00:24 +240 -1: ... prints the platform name when running on multiple platforms
  Expected: contains '[VM]'
    Actual: 'Compiling test.dart...\n'
    '\r00:00 +0: [Chrome] success                                               
                          \r00:00 +1: [Chrome] success                          
                                               \r00:00 +2: [Chrome] success     
                                                                    \r00:00 +2:
All tests passed!                                                               
         \n'
    '\n'
    ''

  package:test                                         expect
  test/runner/browser/compact_reporter_test.dart 39:5  main.<fn>


00:52 +293 -2: test/runner/runner_test.dart: prints help information
  Expected: 'Runs tests in this package.\n'
    '\n'
    'Usage: pub run test:test [files or directories...]\n'
    '\n'
    '-h, --help                Shows this usage information.\n'
    '-n, --name                A substring of the name of the test to run.\n'
    '                          Regular expression syntax is supported.\n'
    '\n'
    '-N, --plain-name          A plain-text substring of the name of the test to
run.\n'
    '-p, --platform            The platform(s) on which to run the tests.\n'
    '                          [vm (default), chrome]\n'
    '\n'
    '    --pub-serve=<port>    The port of a pub serve instance serving
"test/".\n'
    '    --[no-]color          Whether to use terminal colors.\n'
    '                          (auto-detected by default)\n'
    ''
    Actual: 'Runs tests in this package.\n'
    '\n'
    'Usage: pub run test:test [files or directories...]\n'
    '\n'
    '-h, --help                     Shows this usage information.\n'
    '-n, --name                     A substring of the name of the test to
run.\n'
    '                               Regular expression syntax is supported.\n'
    '\n'
    '-N, --plain-name               A plain-text substring of the name of the
test to run.\n'
    '-p, --platform                 The platform(s) on which to run the
tests.\n'
    '                               [vm (default), chrome]\n'
    '\n'
    '-j, --concurrency=<threads>    The number of concurrent test suites run.\n'
    '                               (defaults to 4)\n'
    '\n'
    '    --pub-serve=<port>         The port of a pub serve instance serving
"test/".\n'
    '    --[no-]color               Whether to use terminal colors.\n'
    '                               (auto-detected by default)\n'
    ''
     Which: is different.
  Expected: ...           Shows this ...
    Actual: ...                Shows ...
                          ^
   Differ at offset 111

  package:test                       expect
  test/runner/runner_test.dart 65:5  main.<fn>


00:53 +293 -3: test/runner/runner_test.dart: fails gracefully if an invalid
option is passed
  Expected: 'Could not find an option named "asdf".\n'
    '\n'
    'Usage: pub run test:test [files or directories...]\n'
    '\n'
    '-h, --help                Shows this usage information.\n'
    '-n, --name                A substring of the name of the test to run.\n'
    '                          Regular expression syntax is supported.\n'
    '\n'
    '-N, --plain-name          A plain-text substring of the name of the test to
run.\n'
    '-p, --platform            The platform(s) on which to run the tests.\n'
    '                          [vm (default), chrome]\n'
    '\n'
    '    --pub-serve=<port>    The port of a pub serve instance serving
"test/".\n'
    '    --[no-]color          Whether to use terminal colors.\n'
    '                          (auto-detected by default)\n'
    ''
    Actual: 'Could not find an option named "asdf".\n'
    '\n'
    'Usage: pub run test:test [files or directories...]\n'
    '\n'
    '-h, --help                     Shows this usage information.\n'
    '-n, --name                     A substring of the name of the test to
run.\n'
    '                               Regular expression syntax is supported.\n'
    '\n'
    '-N, --plain-name               A plain-text substring of the name of the
test to run.\n'
    '-p, --platform                 The platform(s) on which to run the
tests.\n'
    '                               [vm (default), chrome]\n'
    '\n'
    '-j, --concurrency=<threads>    The number of concurrent test suites run.\n'
    '                               (defaults to 4)\n'
    '\n'
    '    --pub-serve=<port>         The port of a pub serve instance serving
"test/".\n'
    '    --[no-]color               Whether to use terminal colors.\n'
    '                               (auto-detected by default)\n'
    ''
     Which: is different.
  Expected: ...           Shows this ...
    Actual: ...                Shows ...
                          ^
   Differ at offset 122

  package:test                       expect
  test/runner/runner_test.dart 75:7  main.<fn>.<fn>


01:39 +333 -4: test/runner/pub_serve_test.dart: ... handles pub serve not
running for browser tests
  Expected: 'Failed to load "test/my_test.dart":\n'
    'Error getting http://localhost:54321/my_test.browser_test.dart.js:
Connection refused (errno 111)\n'
    'Make sure "pub serve" is running.\n'
    ''
    Actual: 'Failed to load "test/my_test.dart":\n'
    'Error getting http://localhost:54321/my_test.browser_test.dart.js:
Connection refused (errno 61)\n'
    'Make sure "pub serve" is running.\n'
    ''
     Which: is different.
  Expected: ... ed (errno 111)\nMake ...
    Actual: ... ed (errno 61)\nMake  ...
                          ^
   Differ at offset 130

  package:test                           expect
  test/runner/pub_serve_test.dart 231:5  main.<fn>

[nweiz, 2015-04-06 22:56:43]

Code review changes

[nweiz, 2015-04-06 22:57:15]

On 2015/04/06 22:28:47, kevmoo wrote:
> On 2015/04/06 21:56:44, nweiz wrote:
> 
> Code change LGTM, but I get 4 errors running locally (with bleeding edge SDK
> build)
> 
> 00:24 +240 -1: ... prints the platform name when running on multiple platforms
>   Expected: contains '[VM]'
>     Actual: 'Compiling test.dart...\n'
>     '\r00:00 +0: [Chrome] success                                             
 
>                           \r00:00 +1: [Chrome] success                        
 
>                                                \r00:00 +2: [Chrome] success   
 
>                                                                     \r00:00
+2:
> All tests passed!                                                             
 
>          \n'
>     '\n'
>     ''
> 
>   package:test                                         expect
>   test/runner/browser/compact_reporter_test.dart 39:5  main.<fn>
> 
> 
> 00:52 +293 -2: test/runner/runner_test.dart: prints help information
>   Expected: 'Runs tests in this package.\n'
>     '\n'
>     'Usage: pub run test:test [files or directories...]\n'
>     '\n'
>     '-h, --help                Shows this usage information.\n'
>     '-n, --name                A substring of the name of the test to run.\n'
>     '                          Regular expression syntax is supported.\n'
>     '\n'
>     '-N, --plain-name          A plain-text substring of the name of the test
to
> run.\n'
>     '-p, --platform            The platform(s) on which to run the tests.\n'
>     '                          [vm (default), chrome]\n'
>     '\n'
>     '    --pub-serve=<port>    The port of a pub serve instance serving
> "test/".\n'
>     '    --[no-]color          Whether to use terminal colors.\n'
>     '                          (auto-detected by default)\n'
>     ''
>     Actual: 'Runs tests in this package.\n'
>     '\n'
>     'Usage: pub run test:test [files or directories...]\n'
>     '\n'
>     '-h, --help                     Shows this usage information.\n'
>     '-n, --name                     A substring of the name of the test to
> run.\n'
>     '                               Regular expression syntax is supported.\n'
>     '\n'
>     '-N, --plain-name               A plain-text substring of the name of the
> test to run.\n'
>     '-p, --platform                 The platform(s) on which to run the
> tests.\n'
>     '                               [vm (default), chrome]\n'
>     '\n'
>     '-j, --concurrency=<threads>    The number of concurrent test suites
run.\n'
>     '                               (defaults to 4)\n'
>     '\n'
>     '    --pub-serve=<port>         The port of a pub serve instance serving
> "test/".\n'
>     '    --[no-]color               Whether to use terminal colors.\n'
>     '                               (auto-detected by default)\n'
>     ''
>      Which: is different.
>   Expected: ...           Shows this ...
>     Actual: ...                Shows ...
>                           ^
>    Differ at offset 111
> 
>   package:test                       expect
>   test/runner/runner_test.dart 65:5  main.<fn>
> 
> 
> 00:53 +293 -3: test/runner/runner_test.dart: fails gracefully if an invalid
> option is passed
>   Expected: 'Could not find an option named "asdf".\n'
>     '\n'
>     'Usage: pub run test:test [files or directories...]\n'
>     '\n'
>     '-h, --help                Shows this usage information.\n'
>     '-n, --name                A substring of the name of the test to run.\n'
>     '                          Regular expression syntax is supported.\n'
>     '\n'
>     '-N, --plain-name          A plain-text substring of the name of the test
to
> run.\n'
>     '-p, --platform            The platform(s) on which to run the tests.\n'
>     '                          [vm (default), chrome]\n'
>     '\n'
>     '    --pub-serve=<port>    The port of a pub serve instance serving
> "test/".\n'
>     '    --[no-]color          Whether to use terminal colors.\n'
>     '                          (auto-detected by default)\n'
>     ''
>     Actual: 'Could not find an option named "asdf".\n'
>     '\n'
>     'Usage: pub run test:test [files or directories...]\n'
>     '\n'
>     '-h, --help                     Shows this usage information.\n'
>     '-n, --name                     A substring of the name of the test to
> run.\n'
>     '                               Regular expression syntax is supported.\n'
>     '\n'
>     '-N, --plain-name               A plain-text substring of the name of the
> test to run.\n'
>     '-p, --platform                 The platform(s) on which to run the
> tests.\n'
>     '                               [vm (default), chrome]\n'
>     '\n'
>     '-j, --concurrency=<threads>    The number of concurrent test suites
run.\n'
>     '                               (defaults to 4)\n'
>     '\n'
>     '    --pub-serve=<port>         The port of a pub serve instance serving
> "test/".\n'
>     '    --[no-]color               Whether to use terminal colors.\n'
>     '                               (auto-detected by default)\n'
>     ''
>      Which: is different.
>   Expected: ...           Shows this ...
>     Actual: ...                Shows ...
>                           ^
>    Differ at offset 122
> 
>   package:test                       expect
>   test/runner/runner_test.dart 75:7  main.<fn>.<fn>
> 
> 
> 01:39 +333 -4: test/runner/pub_serve_test.dart: ... handles pub serve not
> running for browser tests
>   Expected: 'Failed to load "test/my_test.dart":\n'
>     'Error getting http://localhost:54321/my_test.browser_test.dart.js:
> Connection refused (errno 111)\n'
>     'Make sure "pub serve" is running.\n'
>     ''
>     Actual: 'Failed to load "test/my_test.dart":\n'
>     'Error getting http://localhost:54321/my_test.browser_test.dart.js:
> Connection refused (errno 61)\n'
>     'Make sure "pub serve" is running.\n'
>     ''
>      Which: is different.
>   Expected: ... ed (errno 111)\nMake ...
>     Actual: ... ed (errno 61)\nMake  ...
>                           ^
>    Differ at offset 130
> 
>   package:test                           expect
>   test/runner/pub_serve_test.dart 231:5  main.<fn>

Fixed the runner changes. The pub_serve change isn't caused by this CL, so I'll
fix it separately.

[kevmoo, 2015-04-06 23:38:36]

LGTM w/ one issue

https://codereview.chromium.org/1062813006/diff/20001/test/runner/runner_test...
File test/runner/runner_test.dart (right):

https://codereview.chromium.org/1062813006/diff/20001/test/runner/runner_test...
test/runner/runner_test.dart:50: (defaults to 6)
not all of us have 12 procs :-)

[nweiz, 2015-04-07 00:14:55]

Code review changes

[nweiz, 2015-04-07 00:15:06]

https://codereview.chromium.org/1062813006/diff/20001/test/runner/runner_test...
File test/runner/runner_test.dart (right):

https://codereview.chromium.org/1062813006/diff/20001/test/runner/runner_test...
test/runner/runner_test.dart:50: (defaults to 6)
On 2015/04/06 23:38:36, kevmoo wrote:
> not all of us have 12 procs :-)

Good catch! Fixed.

[nweiz, 2015-04-07 00:15:17]

Committed patchset #3 (id:40001) manually as
f0d5c2fad78de8346d9b33162f07f6006c3661da (presubmit successful).