Add flutter_engine runtime to the test script so we can run standard dart tests from the test suite…

Add flutter_engine runtime to the test script so we can run standard dart tests from the test suite against the flutter engine on linux/x64

The test suite can be invoked as :
tools/test.py -rflutter --packages=<path_to_packages_dir>/.packages --flutter=<path_to_flutter_engine_workspace>/sky-shell --exclude-suite=pkg

Currently the sky_shell executable does not yet support the proper return
codes:
Success - 0
API error - 253
Compilation error - 254
Unhandled error - 255
Once this is fixed we can start running the tests against our
dart/flutter engine bleeding edge build bot.

R=zra@google.com

Committed: https://github.com/dart-lang/sdk/commit/b583d74608b5d06a8d0f959c2626cb2aa8e8140a

[siva, 2016-12-27 18:52:15]

asiva@google.com changed reviewers:
+ whesse@google.com

[siva, 2017-01-05 17:30:31]

asiva@google.com changed reviewers:
+ zra@google.com

[Bill Hesse, 2017-01-05 17:48:33]

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/runt...
File tools/testing/dart/runtime_configuration.dart (right):

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/runt...
tools/testing/dart/runtime_configuration.dart:262: var args = new List();
I see this is just copied from other cases, but surely

var args = <String>['--non-interactive']..addAll(arguments);

is better.

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
File tools/testing/dart/test_options.dart (right):

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:761: print("-rflutter_engine requires the
flutter engine executable to "
This code seems contradictory to the code in test_suite which provides a default
value for engine.

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
File tools/testing/dart/test_suite.dart (right):

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:211: /// Returns the name of the Sky shell
executable.
Should we use the name "flutter engine" everywhere, instead of both "flutter
engine" and "Sky shell"?

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:214: String skyExecutable =
configuration['engine'];
If the default for this value in test_options was null, then you could just
write
configuration['engine'] ?? 'sky_shell'
Or the default should just be 'sky_shell' in the test_options option
declaration.

This code is in contradiction to the code in test_options which disallows this
case.

[siva, 2017-01-05 20:38:45]

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/runt...
File tools/testing/dart/runtime_configuration.dart (right):

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/runt...
tools/testing/dart/runtime_configuration.dart:262: var args = new List();
On 2017/01/05 17:48:33, Bill Hesse wrote:
> I see this is just copied from other cases, but surely
> 
> var args = <String>['--non-interactive']..addAll(arguments);
> 
> is better.

Done.

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
File tools/testing/dart/test_options.dart (right):

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:761: print("-rflutter_engine requires the
flutter engine executable to "
On 2017/01/05 17:48:33, Bill Hesse wrote:
> This code seems contradictory to the code in test_suite which provides a
default
> value for engine.

Good point, I have taken out the default setting as it is hard for us to auto
compute a default location for the flutter engine binary which is in a different
repo

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
File tools/testing/dart/test_suite.dart (right):

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:211: /// Returns the name of the Sky shell
executable.
On 2017/01/05 17:48:33, Bill Hesse wrote:
> Should we use the name "flutter engine" everywhere, instead of both "flutter
> engine" and "Sky shell"?

Done.

https://codereview.chromium.org/2602783002/diff/40001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:214: String skyExecutable =
configuration['engine'];
On 2017/01/05 17:48:33, Bill Hesse wrote:
> If the default for this value in test_options was null, then you could just
> write
> configuration['engine'] ?? 'sky_shell'
> Or the default should just be 'sky_shell' in the test_options option
> declaration.
> 
> This code is in contradiction to the code in test_options which disallows this
> case.

Removed this default setting, the option is required for running flutter.

[zra, 2017-01-05 20:59:58]

Suggest renaming --engine to --flutter_engine, or maybe --flutter_exe or
--flutter_bin. --engine is maybe too generic.

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
File tools/testing/dart/test_options.dart (right):

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:313: 'engine', 'Path to flutter engine
executable', ['--engine'], [], ''),
maybe 'flutter_engine'?

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:634: 'engine',
flutter_engine?

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:759: if ((config['runtime'] ==
'flutter_engine') && (config['engine'] == '')) {
'engine' -> 'flutter_engine' ?

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:762: "be specified using --engine=");
--flutter_engine ?

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
File tools/testing/dart/test_suite.dart (right):

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:213: // Controlled by user with the option
"--engine".
--flutter_engine

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:214: String flutterExecutable =
configuration['engine'];
'engine' -> 'flutter_engine'

[siva, 2017-01-05 21:27:08]

Description was changed from

==========
Add flutter_engine runtime to the test script so we can run standard dart tests
from the test suite against the flutter engine on linux/x64

The test suite can be invoked as :
tools/test.py -rflutter_engine --packages=<path_to_packages_dir>/.packages
--engine=<path_to_flutter_engine_workspace>/sky-shell --exclude-suite=pkg

Currently the sky_shell executable does not yet support the proper return
codes:
Success - 0
API error - 253
Compilation error - 254
Unhandled error - 255
Once this is fixed we can start running the tests against our
dart/flutter engine bleeding edge build bot.
==========

to

==========
Add flutter_engine runtime to the test script so we can run standard dart tests
from the test suite against the flutter engine on linux/x64

The test suite can be invoked as :
tools/test.py -rflutter --packages=<path_to_packages_dir>/.packages
--flutter=<path_to_flutter_engine_workspace>/sky-shell --exclude-suite=pkg

Currently the sky_shell executable does not yet support the proper return
codes:
Success - 0
API error - 253
Compilation error - 254
Unhandled error - 255
Once this is fixed we can start running the tests against our
dart/flutter engine bleeding edge build bot.
==========

[siva, 2017-01-05 21:27:20]

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
File tools/testing/dart/test_options.dart (right):

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:313: 'engine', 'Path to flutter engine
executable', ['--engine'], [], ''),
On 2017/01/05 20:59:58, zra wrote:
> maybe 'flutter_engine'?

To be consistent with --dart and --dartium, I named it --flutter
It is shorter than flutter_engine, flutter_bin etc.

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:634: 'engine',
On 2017/01/05 20:59:58, zra wrote:
> flutter_engine?

Named it flutter

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:759: if ((config['runtime'] ==
'flutter_engine') && (config['engine'] == '')) {
On 2017/01/05 20:59:58, zra wrote:
> 'engine' -> 'flutter_engine' ?

renamed to flutter

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_options.dart:762: "be specified using --engine=");
On 2017/01/05 20:59:58, zra wrote:
> --flutter_engine ?

renamed to flutter.

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
File tools/testing/dart/test_suite.dart (right):

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:213: // Controlled by user with the option
"--engine".
On 2017/01/05 20:59:58, zra wrote:
> --flutter_engine

Renamed to flutter.

https://codereview.chromium.org/2602783002/diff/60001/tools/testing/dart/test...
tools/testing/dart/test_suite.dart:214: String flutterExecutable =
configuration['engine'];
On 2017/01/05 20:59:58, zra wrote:
> 'engine' -> 'flutter_engine'

Ditto.

[zra, 2017-01-05 21:29:23]

lgtm

[siva, 2017-01-05 22:41:31]

Description was changed from

==========
Add flutter_engine runtime to the test script so we can run standard dart tests
from the test suite against the flutter engine on linux/x64

The test suite can be invoked as :
tools/test.py -rflutter --packages=<path_to_packages_dir>/.packages
--flutter=<path_to_flutter_engine_workspace>/sky-shell --exclude-suite=pkg

Currently the sky_shell executable does not yet support the proper return
codes:
Success - 0
API error - 253
Compilation error - 254
Unhandled error - 255
Once this is fixed we can start running the tests against our
dart/flutter engine bleeding edge build bot.
==========

to

==========
Add flutter_engine runtime to the test script so we can run standard dart tests
from the test suite against the flutter engine on linux/x64

The test suite can be invoked as :
tools/test.py -rflutter --packages=<path_to_packages_dir>/.packages
--flutter=<path_to_flutter_engine_workspace>/sky-shell --exclude-suite=pkg

Currently the sky_shell executable does not yet support the proper return
codes:
Success - 0
API error - 253
Compilation error - 254
Unhandled error - 255
Once this is fixed we can start running the tests against our
dart/flutter engine bleeding edge build bot.

R=zra@google.com

Committed:
https://github.com/dart-lang/sdk/commit/b583d74608b5d06a8d0f959c2626cb2aa8e8140a
==========

[siva, 2017-01-05 22:41:32]

Committed patchset #5 (id:80001) manually as
b583d74608b5d06a8d0f959c2626cb2aa8e8140a (presubmit successful).