Use local pages for webrtc telemetry tests.

tools/perf/page_sets/update_webrtc_cases.py

+# Copyright 2017 The Chromium Authors. All rights reserved.

[nednguyen, 2017/03/24 20:12:37]

Our convention is script has no .py tail. Also please add --help about what this
script does, how to use it. 

+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import urllib2
+
+
+TEST_PAGES_LOCATION_BY_REPO = {
+  'test-pages': [
+      'src/canvas-capture',
+      'src/multiple-peerconnections',
+  ],
+  'samples': [
+      'src/content/datachannel/datatransfer',
+      'src/content/getusermedia/resolution',
+      'src/content/peerconnection/constraints',
+      'src/content/peerconnection/audio',
+  ],
+}
+
+ADDED_SCRIPT_TAGS = (
+    '<script src="%s.js"></script>\n'
+    '<script src="adapter.js"></script>\n'
+    '<script>function trace(arg) { console.log(arg); }</script>\n'
+)
+
+COPYRIGHT_NOTICE = [
+    'Copyright 2017 The Chromium Authors. All rights reserved.\n',
+    'Use of this source code is governed by a BSD-style license that can be\n',
+    'found in the LICENSE file.\n',
+]
+
+JS_COPYRIGHT_NOTICE = ' * '.join(['/*\n'] + COPYRIGHT_NOTICE) + ' */\n'
+HTML_COPYRIGHT_NOTICE = ' * '.join(['<!--\n'] + COPYRIGHT_NOTICE) + '-->\n'
+
+STRIPPED_TAGS = ['<meta', '<link', '<script']

[kjellander_chromium, 2017/03/24 19:51:25]

Do we really want to remove all <script> tags? It seems risky in the future.

Can't you preserve the directory hierarchy on
https://github.com/webrtc/samples/tree/gh-pages/src instead? Like:
tools/perf/page_sets/js/common.js (and thus you can remove line 29).
tools/perf/page_sets/content/datachannel/datatransfer
tools/perf/page_sets/content/getusermedia/resolution
tools/perf/page_sets/content/peerconnection/constraints
tools/perf/page_sets/content/peerconnection/audio

or is there something I'm missing? 
Removing the <link> tags and the <script> tag for ga.js makes sense, but I'm not
sure about the rest.

[ehmaldonado_chromium, 2017/03/24 20:13:53]

I tried to preserve the directory structure, but it looks like levels above the
test page are not accessible (so from peerconnection/getusermedia/index.html I
couldn't access adapter.js

You're right. It also makes sense to copy common.js. about the other scripts,
I'm not sure.

[kjellander_chromium, 2017/03/24 20:18:30]

OK, if there's a limitation in how the files are served and so on, I'm fine with
the change directory structure. I was mostly wondering since it seemed simpler
to just preserve it to a higher degree.

+
+WEBRTC_GITHUB_URL = 'https://github.com/webrtc/'
+ADAPTER_LATEST_URL = 'https://webrtc.github.io/adapter/adapter-latest.js'
+
+
+class TemporaryDirectory(object):
+  def __init__(self):
+    self._closed = False
+    self._name = None
+    self._name = tempfile.mkdtemp()
+  def __enter__(self):
+    return self._name
+  def __exit__(self, exc, value, tb):
+    if self._name and not self._closed:
+      shutil.rmtree(self._name)
+      self._closed = True
+
+
+def CopyJSFile(origin, destination):
+  contents = []
+  with open(origin) as input_file:
+    contents = input_file.readlines()
+
+  contents = [JS_COPYRIGHT_NOTICE] + contents[8:]

[kjellander_chromium, 2017/03/24 19:51:25]

I guess 8 comes from the WebRTC license header which is currently added. It
seems a bit risky but I'm fine since we'll review each update and that tests
should fail badly if the wrong things are removed.

[ehmaldonado_chromium, 2017/03/24 20:13:53]

I could also try to delete the first (multiline) comment. I don't know if it's
safer, though.

[kjellander_chromium, 2017/03/24 20:18:30]

I think it should be, but no big deal since this is just run manually +
reviewed.

+
+  with open(destination, 'w') as output_file:
+    output_file.writelines(contents)
+
+
+def CopyHTMLFile(test_name, origin, destination):
+  contents = []
+  with open(origin) as input_file:
+    for line in input_file:
+      if any(tag in line for tag in STRIPPED_TAGS):

[kjellander_chromium, 2017/03/24 19:51:25]

This seems to assume each of the stripped tags are located on one line only,
which seems like a risky assumption. Can you use multi-line regexp instead?

[ehmaldonado_chromium, 2017/03/24 20:13:53]

Will do.

+        continue
+      if line.strip().startswith('</body>'):
+        contents.append(ADDED_SCRIPT_TAGS % test_name)
+      contents.append(line)
+
+  contents = contents[:1] + [HTML_COPYRIGHT_NOTICE] + contents[8:]
+
+  with open(destination, 'w') as output_file:
+    output_file.writelines(contents)
+
+
+def main():
+  output_dir = sys.argv[1]
+
+  if not os.path.isdir(output_dir):
+    os.makedirs(output_dir)
+
+  with TemporaryDirectory() as temp_dir:
+    for repo_name, test_dirs in TEST_PAGES_LOCATION_BY_REPO.items():
+      p = subprocess.Popen(['git', 'clone', WEBRTC_GITHUB_URL + repo_name],
+                           cwd=temp_dir)
+      p.wait()

[kjellander_chromium, 2017/03/24 19:51:25]

How is the case when git clone fails handled? Does it fail in a way that makes
it easy to figure out what went wrong?

[ehmaldonado_chromium, 2017/03/24 20:13:53]

Yes. Git's output is printed to console.

[kjellander_chromium, 2017/03/24 20:18:30]

That's fine then, since this will be run interactively.

+
+      for test_dir in test_dirs:
+        test_dir = os.path.join(temp_dir, repo_name, test_dir)
+        test_name = os.path.basename(test_dir)
+
+        CopyJSFile(os.path.join(test_dir, 'js', 'main.js'),
+                   os.path.join(output_dir, test_name + '.js'))
+        CopyHTMLFile(test_name, os.path.join(test_dir, 'index.html'),
+                     os.path.join(output_dir, test_name + '.html'))
+
+  response = urllib2.urlopen(ADAPTER_LATEST_URL)
+  adapter_js_contents = JS_COPYRIGHT_NOTICE + response.read()
+
+  with open(os.path.join(output_dir, 'adapter.js'), 'w') as output_file:
+    output_file.write(adapter_js_contents)
+

[kjellander_chromium, 2017/03/24 19:51:25]

+1 blank line

[ehmaldonado_chromium, 2017/03/24 20:13:53]

Will do.

+if __name__ == '__main__':
+  main()