Replace run_selenium.py with Dart script that uses the browser controller for performance tests.

tools/testing/dart/browser_perf_testing/lib/browser_perf_testing.dart

Index: tools/testing/dart/browser_perf_testing/lib/browser_perf_testing.dart
diff --git a/tools/testing/dart/browser_perf_testing/lib/browser_perf_testing.dart b/tools/testing/dart/browser_perf_testing/lib/browser_perf_testing.dart
new file mode 100644
index 0000000000000000000000000000000000000000..d438a5e1cc0bd56bb433009eb868bc8912230441
--- /dev/null
+++ b/tools/testing/dart/browser_perf_testing/lib/browser_perf_testing.dart
@@ -0,0 +1,146 @@
+// Copyright (c) 2014, the Dart project authors.  Please see the AUTHORS file

[kustermann, 2014/02/27 09:37:15]

Please add an import to this file into
"dart/tests/standalone/io/test_harness_analyzer_test.dart".

Since you're using our browser controller, we want to run the analyzer on this
file, to ensure that we don't break it.

[Emily Fortuna, 2014/02/27 19:27:48]

Done.

+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+/// Dart script to launch performance tests without WebDriver/Selenium.
+///
+/// WARNING: Although this is laid out like a package, it is not really a
+/// package since it relies on test.dart files!
+library browser_perf_testing;
+
+import '../../browser_controller.dart';
+import '../../utils.dart';
+import '../../http_server.dart';
+import 'dart:async';
+import 'dart:io';
+import 'dart:convert';
+import 'package:path/path.dart' as path;
+import 'package:args/args.dart' as args_parser;
+
+final String ADDRESS = '127.0.0.1';
+
+/// A map that is passed to the testing framework to specify what ports the
+/// browser controller runs on.
+final Map SERVER_CONFIG = {
+  'test_driver_port': 0,
+  'test_driver_error_port': 0
+};
+
+void main (List<String> args) {
+  var results = _parse_arguments(args);

[kustermann, 2014/02/27 09:37:15]

results -> options/config ?

[Emily Fortuna, 2014/02/27 19:27:48]

Done.

+
+  // Start a server to serve the entire repo: the http server is available on
+  // window.location.port.
+  var servers = new TestingServers(
+      new Path('/Users/efortuna/dart-git2/dart/xcodebuild/ReleaseIA32'),
+      false, results['browser'], path.dirname(path.dirname(path.dirname(
+      path.dirname(path.dirname(path.dirname(Platform.script.path)))))));
+  servers.startServers(ADDRESS).then((_) {
+    _runPerfTests(results, servers);
+  });
+}
+
+/// Helper function to parse the arguments for this file.
+Map _parse_arguments(List<String> args) {

[kustermann, 2014/02/27 09:37:15]

_parse_arguments -> _parseArguments ?

[Emily Fortuna, 2014/02/27 19:27:48]

Thanks. Switching between the Python script and Dart is doing crazy things to my
style!

+  var parser =  new args_parser.ArgParser();
+  parser.addOption('browser', defaultsTo: 'chrome', help: 'Name of the browser'
+      ' to run this test with.');
+  parser.addOption('termination_test_file', defaultsTo:
+      '/root_dart/samples/third_party/dromaeo/dromaeo_end_condition.js',
+      help: 'Path to a javascript file that contains the function '
+      '"testIsComplete" tests whether the performance test has finished '
+      'running. This is in a form that can be served up by '
+      'http_server.dart, so it begins with /root_dart or some other server '
+      'understood prefix.');
+  parser.addOption('test_path', defaultsTo:
+      '/root_dart/samples/third_party/dromaeo/index-js.html?jsANDqueryORjs'
+      'ANDtraverseORjsANDattributes', help: 'Path to the performance test we '
+      'wish to run. This is in a form that can be served up by '
+      'http_server.dart, so it begins with /root_dart or some other server '
+      'understood prefix.');
+  parser.addOption('checked', defaultsTo: false,
+      help: 'Run this test in checked mode.');
+  parser.addFlag('help', abbr: 'h', negatable: false, callback: (help) {
+    if (help) {
+      print(parser.getUsage());
+      exit(0);
+    };
+  });
+  parser.addOption('timeout', defaultsTo: 300,
+      help: 'Maximum amount of time to let a test run, in seconds.');
+  return parser.parse(args);
+}
+
+void _runPerfTests(Map results, TestingServers servers) {
+  var browserName = results['browser'];
+
+  var testRunner = new BrowserTestRunner(SERVER_CONFIG, ADDRESS, browserName, 1,
+      checkedMode: results['checked'],
+      testingServer: new BrowserPerfTestingServer(browserName,
+      results['termination_test_file'], servers.port));
+
+  var url = 'http://$ADDRESS:${servers.port}${results["test_path"]}';
+
+  BrowserTest browserTest = new BrowserTest(url,
+      (BrowserTestOutput output) {
+        var eventQueue = JSON.decode(output.lastKnownMessage);
+        var lastEvent = eventQueue.last;
+        var lines = lastEvent['value'].split('\n');
+        for (var line in lines) {
+          print(line);
+        }
+        testRunner.terminate();
+        servers.stopServers();
+      }, results['timeout']);
+
+  testRunner.start().then((started) {
+    if (started) {
+      testRunner.queueTest(browserTest);
+    } else {
+      print("Issue starting browser test runner $started");
+      exit(1);
+    }
+  });
+}
+
+/// Server for controlling and running performance tests. Note the tests
+/// themselves are served on the local file system (to eliminate any additional
+/// potential sources of lag), but we need a server to communicate when the test
+/// is done.
+class BrowserPerfTestingServer extends BrowserTestingServer {
+  // Path to the script containing the ending condition of the performance test,
+  // in the form of /root_dart, /root_build, or some other, as per the form of
+  // url expected from http_server.dart.
+  String endConditionScript;
+  /// Port number to access the server serving the whole Dart repository.
+  int repoPort;
+
+  BrowserPerfTestingServer(String browserName, this.endConditionScript,
+      this.repoPort) : super(SERVER_CONFIG, ADDRESS,
+      !Browser.BROWSERS_WITH_WINDOW_SUPPORT.contains(browserName));
+
+  /// We create a slightly modified version of the original browser_controller
+  /// driver page.
+  String getDriverPage(String browserId) {
+    var orig = super.getDriverPage(browserId);
+    //TODO(efortuna): Hacky!
+    var insertIndex = orig.indexOf('<script type');
+    var otherInsert = orig.indexOf('} else {',
+        orig.indexOf('if (isStatusUpdate)'));
+    var result = orig.substring(0, insertIndex) + """
+    <!-- To create a performance test, you must write a script that provides an
+  ending condition for the particular test, in the form /root_dart/foo (or
+  whatever base from http_server). // TODO not hard code.-->
+  <script src="http://$localIp:$repoPort$endConditionScript"></script>
+""" + orig.substring(insertIndex, otherInsert) + """
+            if (testIsComplete(msg)) {
+              var obj = new Object();
+              obj['message'] = msg;
+              obj['is_first_message'] = false;
+              obj['is_status_update'] = false;
+              obj['is_done'] = true;
+              window.postMessage(JSON.stringify(obj), '*');
+            }""" + orig.substring(otherInsert);
+    return result;

[kustermann, 2014/02/27 09:37:15]

I don't understand why you want to modify this behavior. I think something like
this should be possible:

var urlToBenchmarkHtml = ...;

var browserTest = new BrowserTest(urlToBenchmarkHtml, (BrowserTestOutput output)
{ 
  if (!output.didTimeout) {
    // Read ${output.message} and extract results from it
  } else {
    // print failure message
  }
});

var runner = new BrowserTestRunner(...);
runner.queueTest(browserTest);

The only thing that is necessary is that the perf test html file (i.e.
[urlToBenchmarkHtml]) behaves according to the protocol described in
tools/testing/dart/test_controller.js:

  5 /*
  6  * The communication protocol between test_controller.js and the driving
  7  * page are JSON encoded messages of the following form:
  8  *   message = {
  9  *      is_first_message: true/false,
 10  *      is_status_update: true/false,
 11  *      is_done: true/false,
 12  *      message: message_content,
 13  *   }
 14  *
 15  * The first message should have [is_first_message] set, the last message
 16  * should have [is_done] set. Status updates should have [is_status_update]
set.
 17  *
 18  * The [message_content] can be be any content. In our case it will a list
of
 19  * events encoded in JSON. See the next comment further down about what an
event
 20  * is.
 21  */

[You **don't** have to use test_controller.js, implementing this communication
protocol is enough.]

So the perf test html only needs to do this:

  window.parent.postMessage(JSON.stringify({
    message: ...,
    is_first_message: ...,
    is_status_update: ...,
    is_done: ...,
  })

I've implemented this on purpose, so the browser controller can be used for
anything (running tests, running benchmarks, other automated things):
- You give it an html file and it will run it.
- The html file can give you status/progress updates if it wants to.
- Once the html file is done, it'll post back a message (can be arbitarily
encoded JSON e.g.) with {is_done: true}
- The browser controller will give you this message.

[Emily Fortuna, 2014/02/27 19:27:48]

The reason I had to modify this is your protocol assumes that no other messages
of any other form are sent. Which in the case of Dromaeo is not the case.
Dromaeo uses message passing within its own test, and the driver page throws a
fit if it receives messages that are not in that form, so I modified the driver
page to not complain. 

Could I have written my own version that doesn't use browser_controller.js? You
bet. But the code is SO SIMILAR with one minor adjustment, it seemed silly to
basically copy/paste everything and change two lines. The performance tests are
also run as correctness tests to ensure that they don't break, so I don't want
to make changes directly to the test itself -- it should be able to run both as
a performance test AND a correctness test. So I add this helper script +
perf_test_controller.js around the original test afterwards, so as not to modify
the original script. 

+  }
+}