Rename test.dart component to specify compiler + runtime.

tools/testing/dart/test_options.dart

 // Copyright (c) 2011, the Dart project authors.  Please see the AUTHORS file
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
 #library("test_options_parser");
 
 #import("dart:io");
 #import("dart:builtin");
 #import("drt_updater.dart");
 
@@ skipping 44 matching lines @@
    */
   TestOptionsParser() {
     _options =
         [ new _TestOptionSpecification(
               'mode',
               'Mode in which to run the tests',
               ['-m', '--mode'],
               ['all', 'debug', 'release'],
               'debug'),
           new _TestOptionSpecification(
-              'component',
-              '''
-Controls how dart code is compiled and executed.
+              'compiler',
+              '''Specify any compilation step (if needed).
 
-   vm: Run dart code on the standalone dart vm.
+   none: Do not compile the Dart code (run native Dart code on the VM).

[ahe, 2012/03/24 13:56:40]

I don't think compiler 'none' should imply running on the VM.

[Emily Fortuna, 2012/03/26 20:52:32]

In what case when the compiler=none would we not be running on a dart VM?

[ahe, 2012/04/13 13:46:20]

Dartium for example.

I'm just saying that, compiler=none should not imply running on the VM. Running
on the VM is an example, for example:

none: Do not compile the Dart code (for example, run native Dart code on the
VM).

[Emily Fortuna, 2012/04/13 17:45:56]

True. This was Ivan's request to make running on the VM the default behavior...

 
-   frog: Compile dart code by running frog on the standalone dart vm, and
-       run the resulting javascript on D8.
+   frog: Compile dart code to JavaScript by running the frog compiler.
 
-   dart2js: Compile dart code by running dart2js on the standalone
-       dart vm, and run the resulting javascript on D8.
+   dart2js: Compile dart code to JavaScript by running dart2js (leg).
 
-   frogsh: Compile dart code by running frog on node.js, and run the
-       resulting javascript on the same instance of node.js.
-
-   dartium: Run dart code in a type="application/dart" script tag in a
-       dartium build of DumpRenderTree.
-
-   frogium: Compile dart code by running frog on the standalone dart vm,
-       and run the resulting javascript in a javascript script tag in
-       a dartium build of DumpRenderTree.
-
-   legium: Compile dart code by running dart2js on the standalone dart
-       vm, and run the resulting javascript in a javascript script tag
-       in a dartium build of DumpRenderTree.
-
-   webdriver: Compile dart code by running frog on the standalone dart vm,
-       and then run the resulting javascript in the browser that is specified
-       by the --browser switch (e.g. chrome, safari, ff, etc.).
-
-   dartc: Run dart code through the dartc static analyzer (does not
-       execute dart code).
+   dartc: Perform static analysis on Dart code by running dartc.
 ''',
-              ['-c', '--component'],
-              ['most', 'vm', 'frog', 'dart2js', 'frogsh', 'dartium',  'frogium',
-               'legium', 'webdriver', 'dartc'],
+              ['-c', '--compiler'],
+              ['none', 'frog', 'dart2js', 'dartc'],

[Siggi Cherem (dart-lang), 2012/03/23 23:57:33]

seems like you still need frogsh? (saw it in status files)

[ahe, 2012/03/24 13:56:40]

I don't think we run frogsh anymore on build bots. I agree with Siggi that
completely removing frogsh could be a separate CL. However, if this CL is
simpler without frogsh, perhaps it is better to remove it now.

Would it help if I prepared a CL to remove frogsh?

[Emily Fortuna, 2012/03/26 20:52:32]

@Peter, SGTM. In this CL I had originally removed all of frogsh, but then I saw
we had one buildbot frogsh-linux-release that seemed to be running frogsh... is
that bot just misnamed at the moment?

+              'none'),
+          new _TestOptionSpecification(
+              'runtime',
+              '''Where the tests should be run.
+    vm: Run Dart code on the standalone dart vm.
+    
+    d8: Run JavaScript from the command line using v8.
+    
+    drt: Run Dart or JavaScript in the headless version of Chrome, 
+        DumpRenderTree.
+    
+    ff or firefox: Run JavaScript in Firefox.
+    
+    chrome: Run JavaScript in Chrome.
+    
+    safari: Run JavaScript in Safari.
+    
+    ie: Run JavaScript in Internet Explorer.
+    
+    opera: Run JavaScript in Opera.
+    
+    none: No runtime (used for dartc static analysis tests).''',

[ahe, 2012/03/24 13:56:40]

for example.

[Emily Fortuna, 2012/03/26 20:52:32]

Done.

+              ['-r', '--runtime'],
+              ['vm', 'd8', 'drt', 'ff', 'firefox', 'chrome', 
+              'safari', 'ie', 'opera', 'none'],
               'vm'),
           new _TestOptionSpecification(
               'arch',
               'The architecture to run tests for',
               ['-a', '--arch'],
               ['all', 'ia32', 'x64', 'simarm'],
               'ia32'),
           new _TestOptionSpecification(
               'system',
               'The operating system to run tests on',
@@ skipping 84 matching lines @@
               [],
               ''),
           new _TestOptionSpecification(
               'time',
               'Print timing information after running tests',
               ['--time'],
               [],
               false,
               'bool'),
           new _TestOptionSpecification(
-              'browser',
-              'Web browser to use on webdriver tests',
-              ['-b', '--browser'],
-              ['ff', 'chrome', 'safari', 'ie', 'opera'],
-              'chrome'),
-          new _TestOptionSpecification(
-              'frog',
-              'Path to frog executable',
-              ['--frog'],
-              [],
-              ''),
-          new _TestOptionSpecification(
               'drt',
               'Path to DumpRenderTree executable',
               ['--drt'],
               [],
               ''),
           new _TestOptionSpecification(
-              'froglib',
-              'Path to frog library',
-              ['--froglib'],
-              [],
-              ''),
-          new _TestOptionSpecification(
               'noBatch',
               'Do not run browser tests in batch mode',
               ['-n', '--nobatch'],
               [],
               false,
               'bool')];
   }
 
 
   /**
@@ skipping 119 matching lines @@
    * into a list of configurations with exactly one value per key.
    */
   List<Map> _expandConfigurations(Map configuration) {
     // Expand the pseudo-values such as 'all'.
     if (configuration['arch'] == 'all') {
       configuration['arch'] = 'ia32,x64';
     }
     if (configuration['mode'] == 'all') {
       configuration['mode'] = 'debug,release';
     }
-    if (configuration['component'] == 'most') {
-      configuration['component'] = 'vm,dartc';
-    }
     if (configuration['valgrind']) {
       // TODO(ager): Get rid of this when there is only one checkout and
       // we don't have to special case for the runtime checkout.
       File valgrindFile = new File('runtime/tools/valgrind.py');
       if (!valgrindFile.existsSync()) {
         valgrindFile = new File('../runtime/tools/valgrind.py');
       }
       String valgrind = valgrindFile.fullPathSync();
       configuration['special-command'] = 'python -u $valgrind @';
     }
 
     // Use verbose progress indication for verbose output unless buildbot
     // progress indication is requested.
     if (configuration['verbose'] && configuration['progress'] != 'buildbot') {
       configuration['progress'] = 'verbose';
     }
 
     // Create the artificial 'unchecked' options that test status files
     // expect.
     configuration['unchecked'] = !configuration['checked'];
     configuration['host_unchecked'] = !configuration['host_checked'];
 
+    if (configuration['runtime'] == 'firefox') {
+      configuration['runtime'] == 'ff';
+    }
+
     // Expand the test selectors into a suite name and a simple
     // regular expressions to be used on the full path of a test file
     // in that test suite. If no selectors are explicitly given use
     // the default suite patterns.
     var selectors = configuration['selectors'];
     if (selectors is !Map) {
       if (selectors == null) {
         selectors = new List.from(defaultTestSelectors);
       }
       Map<String, RegExp> selectorMap = new Map<String, RegExp>();
@@ skipping 13 matching lines @@
           print("Error: '$suite/$pattern'.  Only one test selection" +
                 " pattern is allowed to start with '$suite/'");
           exit(1);
         }
         selectorMap[suite] = new RegExp(pattern);
       }
       configuration['selectors'] = selectorMap;
     }
 
     // Expand the architectures.
-    var archs = configuration['arch'];
-    if (archs.contains(',')) {
-      var result = new List<Map>();
-      for (var arch in archs.split(',')) {
-        var newConfiguration = new Map.from(configuration);
-        newConfiguration['arch'] = arch;
-        result.addAll(_expandConfigurations(newConfiguration));
-      }
-      return result;
+    if (configuration['arch'].contains(',')) {
+      return _expandHelper('arch', configuration);
     }
 
     // Expand modes.
-    var modes = configuration['mode'];
-    if (modes.contains(',')) {
-      var result = new List<Map>();
-      for (var mode in modes.split(',')) {
-        var newConfiguration = new Map.from(configuration);
-        newConfiguration['mode'] = mode;
-        result.addAll(_expandConfigurations(newConfiguration));
-      }
-      return result;
+    if (configuration['mode'].contains(',')) {
+      return _expandHelper('mode', configuration);
+    }
+        
+    // Expand compilers.
+    if (configuration['compiler'].contains(',')) {
+      return _expandHelper('compiler', configuration);
     }
 
-    // Expand components.
-    var components = configuration['component'];
-    if (components.contains(',')) {
-      var result = new List<Map>();
-      for (var component in components.split(',')) {
-        var newConfiguration = new Map.from(configuration);
-        newConfiguration['component'] = component;
-        result.addAll(_expandConfigurations(newConfiguration));
-      }
-      return result;
+    // Expand runtimes.
+    var runtimes = configuration['runtime'];
+    if (runtimes.contains(',')) {
+      return _expandHelper('runtime', configuration);

[Siggi Cherem (dart-lang), 2012/03/23 23:57:33]

could we add a function (or a TODO) that checks that the blend of configurations
makes sense and issue a warning/error if they don't? For example:

-c dartc -r drt would be invalid.

I prefer to issue a warning, so that if you specify many combinations together
in the command line, e.g.: '-c frog,dartc -r none,drt', then you see some
warnings but all other valid tests are run:
 warning: frog_none is not valid
 warning: dartc_drt is not valid
 running tests for frog_drt
 running tests for dartc_none

[Emily Fortuna, 2012/03/26 20:52:32]

Done.

[Siggi Cherem (dart-lang), 2012/03/26 22:28:39]

Nice, thanks!

     } else {
-      // All components eventually go through this path, after expansion.
-      if (DumpRenderTreeUpdater.componentRequiresDRT(components)) {
+      // All runtimes eventually go through this path, after expansion.
+      if (runtimes == 'drt') {
         DumpRenderTreeUpdater.update();
       }
     }
 
-    // Adjust default timeout based on mode and component.
+    // Adjust default timeout based on mode and runtime.
     if (configuration['timeout'] == -1) {
       var timeout = 60;
-      switch (configuration['component']) {
-        case 'dartc':
-        case 'dartium':
-        case 'frogium':
-        case 'legium':
-        case 'webdriver':
+      switch (configuration['runtime']) {
+        case 'none': // none = dartc static analysis

[ahe, 2012/03/24 13:56:40]

I would prefer that none does not imply dartc.

[Emily Fortuna, 2012/03/26 20:52:32]

Done.

+        case 'drt':
+        case 'ie':
+        case 'ff':
+        case 'chrome':
+        case 'safari':
+        case 'opera':
           timeout *= 4;
           break;
-        case 'dart2js':
-        case 'frog':
+        case 'd8':

[ahe, 2012/03/24 13:56:40]

It has nothing to do with d8, the problem is the compiler takes too long when
running on the debug VM or in host_checked mode.

So I think you need to look at the compiler (I would like to be able to test
dart2js with runtime == none).

[Emily Fortuna, 2012/03/26 20:52:32]

Done.

           if (configuration['mode'] == 'debug') {
             timeout *= 8;
           }
           if (configuration['host_checked']) {
             timeout *= 16;
           }
           break;
         default:
           if (configuration['mode'] == 'debug') {
             timeout *= 2;
           }
           break;
       }
       configuration['timeout'] = timeout;
     }
 
     return [configuration];
   }
 
+  /** 
+   * Helper for _expandConfigurations. Creates a new configuration and adds it
+   * to a list, for use in a case when a particular configuration has multiple
+   * results (separated by a ','.

[ahe, 2012/03/24 13:56:40]

Missing close parens.

[Emily Fortuna, 2012/03/26 20:52:32]

Done.

+   * Arguments:
+   * option: The particular test option we are expanding.
+   * configuration: The map containing all test configuration information
+   * specified.
+   */
+  List<Map> _expandHelper(String option, Map configuration) {
+    var result = new List<Map>();
+    var configs = configuration[option]; 
+    for (var config in configs.split(',')) {
+      var newConfiguration = new Map.from(configuration);
+      newConfiguration[option] = config;
+      result.addAll(_expandConfigurations(newConfiguration));
+    }
+    return result;
+  }
+
 
   /**
    * Print out usage information.
    */
   void _printHelp() {
     print('usage: dart test.dart [options]\n');
     print('Options:\n');
     for (var option in _options) {
       print('${option.name}: ${option.description}.');
       for (var name in option.keys) {
@@ skipping 35 matching lines @@
         return option;
       }
     }
     print('Unknown test option $name');
     exit(1);
   }
 
 
   List<_TestOptionSpecification> _options;
 }