Android: first step into making "perf tests" reuse the functional test infra.

build/android/pylib/perf/setup.py

Index: build/android/pylib/perf/setup.py
diff --git a/build/android/pylib/perf/setup.py b/build/android/pylib/perf/setup.py
new file mode 100644
index 0000000000000000000000000000000000000000..f36959f3d633a8062fdc9fbbf21fc39a85648900
--- /dev/null
+++ b/build/android/pylib/perf/setup.py
@@ -0,0 +1,78 @@
+# Copyright 2013 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Generates test runner factory and tests for performance tests."""
+
+import json
+import logging
+import os
+import signal
+import test_runner
+
+from pylib import android_commands
+from pylib import cmd_helper
+from pylib import forwarder
+from pylib import ports
+
+
+def _KillPendingServers():
+  for retry in range(5):
+    for server in ['lighttpd', 'web-page-replay']:
+      pids = cmd_helper.GetCmdOutput(['pgrep', '-f', server])

[frankf, 2013/08/12 23:09:52]

Can you use psutil: http://stackoverflow.com/a/4230226

[bulach, 2013/08/13 08:58:19]

Done.

+      pids = [pid.strip() for pid in pids.split('\n') if pid.strip()]
+      for pid in pids:
+        try:
+          logging.warning('Killing %s %s', server, pid)
+          os.kill(int(pid), signal.SIGQUIT)
+        except Exception as e:
+          logging.warning('Failed killing %s %s %s', server, pid, e)
+  # Restart the adb server with taskset to set a single CPU affinity.
+  cmd_helper.RunCmd(['adb', 'kill-server'])
+  cmd_helper.RunCmd(['taskset', '-c', '0', 'adb', 'start-server'])

[frankf, 2013/08/12 23:09:52]

Are we going to eventually do this in general for other test types?

[bulach, 2013/08/13 08:58:19]

potentially, yes.. but maybe best to start here, then generalize it later?

+  cmd_helper.RunCmd(['taskset', '-c', '0', 'adb', 'root'])
+  i = 1
+  while not android_commands.GetAttachedDevices():
+    time.sleep(i)
+    i *= 2
+    if i > 10:
+      break
+  # Reset the test port allocation. It's important to do it before starting
+  # to dispatch any step.
+  if not ports.ResetTestServerPortAllocation():

[frankf, 2013/08/12 23:09:52]

Currently we call this from bunch of places:
For gtests: gtest/setup.py
For instrumentation tests: test_runner.py
And now here for perf.

Can we unify these into one call in test_runner.py?

[bulach, 2013/08/13 08:58:19]

since there's already two other places, would you mind this refactoring as a
follow up?
reason being that this patch will only run downstream, it'd be nice to ensure
this biggie won't affect any other bot... then shortly after this patch
"sticks", I'm happy to do the refactoring across the board.. wdyt?

+    raise Exception('Failed to reset test server port.')
+
+  forwarder.Forwarder.UseMultiprocessing()

[frankf, 2013/08/12 23:09:52]

This is now multi-threading. Can we get rid of this?

[bulach, 2013/08/13 08:58:19]

unfortunately not, it's multi-threading only on the first level.. :)
internally, test_runner.py is then calling subprocess.Popen to run the commands
(i.e., telemetry, or downstream startup tests), which then need this envvar to
setup the forwarder accordingly.

+
+
+def Setup(test_options):
+  """Create and return the test runner factory and tests.
+
+  Args:
+    test_options: A PerformanceOptions object.
+
+  Returns:
+    A tuple of (TestRunnerFactory, tests).
+  """
+  # Just print the results from a single previously executed step.
+  if test_options.print_step:
+    def TestRunnerPrinterFactory(device, shard_index):
+      return test_runner.TestPrinter(test_options, device, shard_index)

[frankf, 2013/08/13 00:43:12]

Also, it's really weird to have the printing logic use the test runner as it's
purely a host-side operation. Can you let _RunPerfTests in test_runner.py decide
what to do?

[bulach, 2013/08/13 08:58:19]

yeah, I fully agree this is a bizarre shoehorning... :(
moving it up to _RunPerfTests is a good idea! done.

+    return (TestRunnerPrinterFactory, [test_options.print_step])
+
+  # Before running the tests, kill any leftover server.
+  _KillPendingServers()
+
+  with file(test_options.steps, 'r') as f:
+    tests = json.load(f)
+
+  flaky_steps = []
+  if test_options.flaky_steps:
+    with file(test_options.flaky_steps, 'r') as f:
+      flaky_steps = json.load(f)
+
+  def TestRunnerFactory(device, shard_index):
+    return test_runner.TestRunner(
+        test_options, device, shard_index, tests, flaky_steps)
+
+  return (TestRunnerFactory, sorted(tests.keys()))

[frankf, 2013/08/12 23:09:52]

Any reasons to sort the tests given you don't sort the devices anymore?

[bulach, 2013/08/13 08:58:19]

that's actually a good point. I added "sorted" in
test_dispatcher.py::_GetAttachedDevices.. I think it won't hurt, and will keep
things a bit more deterministic.