tools: Add a tool to load pages on a device, and pull the predictor database.

tools/resource_prefetch_predictor/generate_database.py

+#!/usr/bin/python
+#
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Loads a set of web pages several times on a device, and extracts the
+predictor database.
+"""
+
+import logging
+import optparse
+import os
+import subprocess
+import sys
+import time
+
+_SRC_PATH = os.path.abspath(os.path.join(
+    os.path.dirname(__file__), os.pardir, os.pardir))
+sys.path.append(os.path.join(_SRC_PATH, 'third_party', 'catapult', 'devil'))
+
+from devil.android import device_utils
+from devil.android import flag_changer
+from devil.android.constants import chrome
+from devil.android.sdk import intent
+
+sys.path.append(os.path.join(_SRC_PATH, 'build', 'android'))
+import devil_chromium
+
+sys.path.append(os.path.join(_SRC_PATH, 'tools', 'android', 'loading'))
+import device_setup

[pasko, 2016/11/10 16:02:00]

do we need this dependency?

[Benoit L, 2016/11/14 12:49:25]

Done.

+
+
+_PAGE_LOAD_TIMEOUT = 20
+
+# Local build of Chrome (not Chromium).
+_PACKAGE_INFO = chrome.PACKAGE_INFO['chrome']
+
+# Command line arguments for Chrome.
+_CHROME_FLAGS = [
+    # Disable backgound network requests that may pollute WPR archive, pollute
+    # HTTP cache generation, and introduce noise in loading performance.

[pasko, 2016/11/10 16:02:00]

this is getting repeated, should some sort of common function in
tools/android/loading return the list?

[Benoit L, 2016/11/14 12:49:25]

Done.

+    '--disable-background-networking',
+    '--disable-default-apps',
+    '--no-proxy-server',
+    '--safebrowsing-disable-auto-update',
+
+    # Disables actions that chrome performs only on first run or each launches,
+    # which can interfere with page load performance, or even block its
+    # execution by waiting for user input.
+    '--disable-fre',
+    '--no-default-browser-check',
+    '--no-first-run',
+
+    '--speculative-resource-prefetching=learning',
+]
+
+
+def _CreateOptionParser():
+  parser = optparse.OptionParser(
+      description='Loads a set of web pages several times on a device, and '
+      'extracts the predictor database.')
+  parser.add_option('--device', help='Device ID')
+  parser.add_option('--urls_filename', help='File containing a list of URLs '
+                    '(one per line). URLs can be repeated.')
+  parser.add_option('--output_filename', help='File to store the database in.')
+  parser.add_option('--repeats', help='Number of times each URL is loaded.',

[pasko, 2016/11/10 16:02:01]

nit: s/repeats/url-repeat/ sounds more consistent with similar params in
telemetry

also, ambiguity: this seems to be about repeating working on each line in the
file, not about times to load each unique url

[Benoit L, 2016/11/14 12:49:25]

Done.

+                    default=3)
+  return parser
+
+
+def _FindDevice(device_id):
+  """Returns a device matching |device_id| or the first one if None, or None."""
+  devices = device_utils.DeviceUtils.HealthyDevices()
+  if device_id is None:
+    return devices[0]
+  matching_devices = [d for d in devices if str(d) == device_id]
+  if not matching_devices:
+    return None
+  return matching_devices[0]
+
+
+def _ResetChromeLocalState(device, package_name):
+  """Removes the Chrome Profile and the various disk caches."""
+  profile_dirs = ['app_chrome/Default', 'cache', 'app_chrome/ShaderCache',

[pasko, 2016/11/10 16:02:00]

Do you want to avoid dependency on tools/android/loading/controller.py? Why?

[Benoit L, 2016/11/14 12:49:25]

Acknowledged.

+                  'app_tabs']
+  cmd = ['rm', '-rf']
+  cmd.extend('/data/user/0/%s/%s' % (package_name, d) for d in profile_dirs)
+  device.adb.Shell(subprocess.list2cmdline(cmd))
+
+
+def _Setup(device):
+  """Configures the device."""
+  if not device.HasRoot():
+    device.EnableRoot()
+  changer = flag_changer.FlagChanger(
+      device, '/data/local/tmp/chrome-command-line')

[pasko, 2016/11/10 16:02:00]

the controller.py is doing it with /data/local/chrome-command-line. Do they both
work?

[Benoit L, 2016/11/14 12:49:25]

Done.

+  changer.ReplaceFlags(_CHROME_FLAGS)

[pasko, 2016/11/10 16:02:00]

I like the FlagReplacer contextmanager, getting the device stuck with some
random flags file because of an exception is .. a thing from the past .. the
past when we loved Telemetry

[Benoit L, 2016/11/14 12:49:25]

Acknowledged.

+  device.ForceStop(_PACKAGE_INFO.package)
+  _ResetChromeLocalState(device, _PACKAGE_INFO.package)
+
+
+def _Go(device, urls_filename, output_filename, repeats):
+  urls = []
+  with open(urls_filename) as f:
+    urls = [line.strip() for line in f.readlines()]
+
+  for repeat in range(repeats):
+    logging.info('Repeat #%d', repeat)
+    for url in urls:
+      logging.info('\tURL: %s', url)
+      view_intent = intent.Intent(action='android.intent.action.VIEW',
+                                  package=_PACKAGE_INFO.package,
+                                  activity=_PACKAGE_INFO.activity,
+                                  data=url)
+      device.StartActivity(view_intent, blocking=True)
+      time.sleep(_PAGE_LOAD_TIMEOUT)
+
+  # Get the file.
+  device.ForceStop(_PACKAGE_INFO.package)
+  database_filename = (
+      '/data/user/0/%s/app_chrome/Default/Network Action Predictor' %
+      _PACKAGE_INFO.package)
+  device.PullFile(database_filename, output_filename)
+
+
+def main():
+  logging.basicConfig(level=logging.INFO)
+  parser = _CreateOptionParser()
+  options, _ = parser.parse_args()
+  devil_chromium.Initialize()
+  device = _FindDevice(options.device)
+  if device is None:
+    logging.error('No suitable device.')

[pasko, 2016/11/10 16:02:00]

nit: would be more informative to:

could not find device: options.device

[Benoit L, 2016/11/14 12:49:25]

Done.

+    sys.exit(1)
+  _Setup(device)
+  _Go(device, options.urls_filename, options.output_filename,
+      int(options.repeats))
+
+
+if __name__ == '__main__':
+  main()