[Offline Pages] Script for running evaluation tests.

chrome/browser/android/offline_pages/evaluation/run_offline_page_evaluation_test.py

Index: chrome/browser/android/offline_pages/evaluation/run_offline_page_evaluation_test.py
diff --git a/chrome/browser/android/offline_pages/evaluation/run_offline_page_evaluation_test.py b/chrome/browser/android/offline_pages/evaluation/run_offline_page_evaluation_test.py
new file mode 100755
index 0000000000000000000000000000000000000000..828d000eab062808bf09574a8eb25a3b8c98ee33
--- /dev/null
+++ b/chrome/browser/android/offline_pages/evaluation/run_offline_page_evaluation_test.py
@@ -0,0 +1,129 @@
+#!/usr/bin/python2
+#
+# Copyright (c) 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+#
+#
+# This script is used to run tests for SavePageLater evaluation.

[Pete Williamson, 2016/11/02 20:30:29]

Are the instructions in "Instructions to run
OfflinePageSavePageLaterEvaluationTest" updated to show how to use the script?

[romax, 2016/11/03 01:46:50]

Not yet, I'm planning to update the instructions after the patch lands.

+
+import argparse
+import os
+import shutil
+import subprocess
+import sys
+
+DEFAULT_URL_TIMEOUT = 180

[dougarnett, 2016/11/02 19:34:15]

too small for 2g, consider upto 8 minutes instead (or we can split difference
for default at 5 or 6 minutes)

[Pete Williamson, 2016/11/02 20:30:29]

Should there be a way to use the current timeout policy of the request
coordinator instead of overriding it?

Our (soon to be current) timeout policy is 3 tries, and tries are 8 minutes for
an "immediate" try, 3 minutes for a "scheduled" try.

(so we might allow as much as 24 minutes for an "immediate" try, and 9 minutes
for a scheduled try).

[romax, 2016/11/03 01:46:49]

Changed default to 480sec. I think it makes sense to use policy to control the
test time limit when using test scheduler. However when it comes to test with
actual scheduling, I prefer to have a explicit time limit for the test. I'm
marking it as TODO.

+DEFAULT_USER_REQUEST = False
+DEFAULT_USE_TEST_SCHEDULER = False
+DEFAULT_VERBOSE = False
+CONFIG_FILENAME = 'test_config'
+CONFIG_TEMPLATE = """\
+TimeoutPerUrlInSeconds = {timeout_per_url_in_seconds}
+IsUserRequested = {is_user_requested}
+UseTestScheduler = {use_test_scheduler}
+"""
+

[Pete Williamson, 2016/11/02 20:30:29]

Maybe you could provice a bit of guidance here in how to use the script.  An
example invocation with the flags most people will need would be good.

[romax, 2016/11/03 01:46:50]

Should I attach the link to the instructions? But an example has been added at
the top of the script.

[Pete Williamson, 2016/11/03 18:21:11]

It is better to have instructions here, since Chromium is open source, and
coders won't be able to look at private docs.

[romax, 2016/11/03 20:17:20]

Done.

+
+def main(args):
+  # Setting up the argument parser.
+  parser = argparse.ArgumentParser()
+  parser.add_argument(
+      '--output-directory',
+      dest='output_dir',
+      help='Directory for output, default is ~/offline_eval_output/')

[Pete Williamson, 2016/11/02 20:30:29]

Thanks for adding the help!

[romax, 2016/11/03 01:46:50]

:)

+  parser.add_argument(
+      '--url-timeout',
+      type=int,
+      dest='url_timeout',
+      help='Time out per url, in seconds.')

[Pete Williamson, 2016/11/02 20:30:29]

Maybe mention default here, and in other help items?

[romax, 2016/11/03 01:46:49]

Done.

+  parser.add_argument(
+      '--user-request',
+      dest='user_request',
+      action='store_true',
+      help='Test as user-requested urls.')

[Pete Williamson, 2016/11/02 20:30:29]

--user-request should default to true, does it?  (I see
DEFAULT_USER_REQUEST=False further up in the file)

[romax, 2016/11/03 01:46:49]

Done.

+  parser.add_argument(
+      '--use-test-scheduler',
+      dest='use_test_scheduler',
+      action='store_true',
+      help='Start processing immediately, no using GCMNetworkManager.')
+  parser.add_argument(
+      '-v',
+      '--verbose',
+      dest='verbose',
+      action='store_true',
+      help='Make test runner verbose.')
+  parser.add_argument('build_output_dir', help='Path to build directory.')
+  parser.add_argument(
+      'test_urls_file', help='Path to input file with urls to be tested.')
+  parser.set_defaults(
+      output_dir=os.path.expanduser('~/offline_eval_output'),
+      url_timeout=DEFAULT_URL_TIMEOUT,
+      user_request=DEFAULT_USER_REQUEST,
+      user_test_scheduler=DEFAULT_USE_TEST_SCHEDULER,
+      verbose=DEFAULT_VERBOSE)
+
+  # Get the arguments and several paths.
+  options, extra_args = parser.parse_known_args(args)
+
+  if extra_args:
+    print 'Unknown args: ' + ', '.join(
+        extra_args) + '. Please check and run again.'
+    return
+
+  build_dir_path = os.path.abspath(
+      os.path.join(os.getcwd(), options.build_output_dir))
+  test_runner_path = os.path.join(build_dir_path,
+                                  'bin/run_chrome_public_test_apk')
+  config_output_path = os.path.join(options.output_dir, CONFIG_FILENAME)
+
+  # Create the output directory for results, and have a copy of test config
+  # there.
+  if not os.path.exists(options.output_dir):
+    print 'Creating output directory for results... ' + options.output_dir
+    os.makedirs(options.output_dir)
+  with open(config_output_path, 'w') as config:
+    config.write(
+        CONFIG_TEMPLATE.format(
+            timeout_per_url_in_seconds=options.url_timeout,
+            is_user_requested=options.user_request,
+            use_test_scheduler=options.use_test_scheduler))
+
+  print 'Uploading config file and input file onto the device.'
+  subprocess.call(
+      ['adb', 'push', config_output_path, '/sdcard/paquete/test_config'])
+  subprocess.call([
+      'adb', 'push', options.test_urls_file,
+      '/sdcard/paquete/offline_eval_urls.txt'

[Pete Williamson, 2016/11/02 20:30:29]

What happens on a device with no SD card, like the nexus?

[romax, 2016/11/03 01:46:50]

the 'virtual' external storage would be mounted to /sdcard/ on nexus 7, (but i'm
not sure on other devices). However I found an 'adb' way to get the external
storage.

[Pete Williamson, 2016/11/03 18:21:11]

So will this work on my Nexus 5X and Nexus 6 devices which don't have an SD
card?

[romax, 2016/11/03 20:17:20]

Done.
Per discussion it should work, but I'd like to get one and try :)

+  ])
+  print 'Start running test...'
+
+  # Run test
+  test_runner_cmd = [
+      test_runner_path, '-f',
+      'OfflinePageSavePageLaterEvaluationTest.testFailureRateWithTimeout'
+  ]
+  if options.verbose:
+    test_runner_cmd += ['-v']
+  subprocess.call(test_runner_cmd)
+
+  print 'Fetching results from device...'
+  archive_dir = os.path.join(options.output_dir, 'archives/')
+  if os.path.exists(archive_dir):
+    shutil.rmtree(archive_dir)
+  subprocess.call([
+      'adb', 'pull', '/data/data/org.chromium.chrome/app_chrome/'
+      'Default/Offline Pages/archives', archive_dir
+  ])
+  subprocess.call([
+      'adb', 'pull', '/sdcard/paquete/offline_eval_results.txt',
+      options.output_dir
+  ])
+  subprocess.call([
+      'adb', 'pull', '/sdcard/paquete/offline_eval_logs.txt', options.output_dir
+  ])
+  print 'Test finished!'
+
+
+if __name__ == '__main__':
+  sys.exit(main(sys.argv[1:]))