[WPT Export] Require GitHub API token in WPT remote origin URL

third_party/WebKit/Tools/Scripts/webkitpy/w3c/local_wpt.py

 # Copyright 2016 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 """A utility class for interacting with a local checkout of the Web Platform Tests."""
 
 import logging
 
 from webkitpy.common.system.executive import ScriptError
 from webkitpy.w3c.chromium_commit import ChromiumCommit
-from webkitpy.w3c.common import WPT_GH_REPO_URL, CHROMIUM_WPT_DIR
+from webkitpy.w3c.common import WPT_GH_REPO_URL_TEMPLATE, CHROMIUM_WPT_DIR
 
 
 _log = logging.getLogger(__name__)
 
 
 class LocalWPT(object):
 
-    def __init__(self, host, path='/tmp/wpt'):
+    def __init__(self, host, gh_token, path='/tmp/wpt'):

[qyearsley, 2017/02/28 18:30:04]

LocalWPT is used in test_importer, which I believe doesn't need a token -- so
here, gh_token should perhaps be a named optional argument with a default value
of None?

[jeffcarp, 2017/02/28 18:53:38]

sgtm

         """
         Args:
             host: A Host object.
             path: Optional, the path to the web-platform-tests repo.
                 If this directory already exists, it is assumed that the
                 web-platform-tests repo is already checked out at this path.
         """
         self.host = host
         self.path = path
+        self.gh_token = gh_token
         self.branch_name = 'chromium-export-try'
 
     def fetch(self):
         if self.host.filesystem.exists(self.path):
             _log.info('WPT checkout exists at %s, fetching latest', self.path)
             self.run(['git', 'fetch', 'origin'])
             self.run(['git', 'checkout', 'origin/master'])
         else:
-            _log.info('Cloning %s into %s', WPT_GH_REPO_URL, self.path)
-            self.host.executive.run_command(['git', 'clone', WPT_GH_REPO_URL, self.path])
+            _log.info('Cloning GitHub w3c/web-platform-tests into %s', self.path)
+            remote_url = WPT_GH_REPO_URL_TEMPLATE.format(self.gh_token)
+            self.host.executive.run_command(['git', 'clone', '--depth=10', remote_url, self.path])

[qyearsley, 2017/02/28 18:30:04]

Why depth = 10? What if the test that we want to upstream is nested really
deeply, like 11 or 12 directories deep?

[jeffcarp, 2017/02/28 18:53:38]

--depth in this case refers to how many commits back we want to clone (e.g. this
only clones the repository and its most recent 10 commits), but your comment
made me realize that this won't work for a similar reason, because the commit
containing Cr-Commit-Position might be more than 10 commits back.

I added --depth originally to lighten the load since we're cloning the GitHub
repo from scratch every 5-10 minutes or so (since the bot gets torn down every
time). I'll remove that flag and think of other ways to lighten the load.

[qyearsley, 2017/02/28 18:58:04]

Ah, I see, that's what depth means :-)

Maybe another depth might be reasonable, as an optimization? Like, there should
be no need to go back further than a few months of commits, probably?

 
     def run(self, command, **kwargs):
         """Runs a command in the local WPT directory."""
         return self.host.executive.run_command(command, cwd=self.path, **kwargs)
 
     def most_recent_chromium_commit(self):
         """Finds the most recent commit in WPT with a Chromium commit position."""
         wpt_commit_hash = self.run(['git', 'rev-list', 'HEAD', '-n', '1', '--grep=Cr-Commit-Position'])
         if not wpt_commit_hash:
             return None, None
@@ skipping 76 matching lines @@
 
     def commits_behind_master(self, commit):
         """Returns the number of commits after the given commit on origin/master.
 
         This doesn't include the given commit, and this assumes that the given
         commit is on the the master branch.
         """
         return len(self.run([
             'git', 'rev-list', '{}..origin/master'.format(commit)
         ]).splitlines())