telemetry: Create a helper class to quickly extend profiles.

tools/perf/profile_creators/fast_navigation_profile_extender.py

+# Copyright 2014 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+import time
+
+from telemetry.core import browser_finder
+from telemetry.core import browser_finder_exceptions
+from telemetry.core import exceptions
+from telemetry.core import util
+
+
+class FastNavigationProfileExtender(object):
+  """
+  This class creates or extends an existing profile by performing a set of tab
+  navigations in large batches. This is accomplished by opening a large number
+  of tabs, simultaneously navigating all the tabs, and then waiting for all the
+  tabs to load. This provides two benefits:
+    - Takes advantage of the high number of logical cores on modern CPUs.
+    - The total time spent waiting for navigations to time out scales linearly
+      with the number of batches, but does not scale with the size of the
+      batch.
+  """
+  def __init__(self):
+    super(FastNavigationProfileExtender, self).__init__()
+
+    # A reference to the browser that will be performing all of the tab
+    # navigations.
+    self._browser = None
+
+    # A static copy of the urls that this class is going to navigate to.
+    self._navigation_urls = self.NavigationUrls()

[nednguyen, 2015/02/09 21:35:02]

def GetUrlsToNavigate()

[erikchen, 2015/02/10 00:45:02]

Done.

+
+    # The number of tabs to use.
+    self._NUM_TABS = 15
+
+    # The number of pages to load in parallel.
+    self._NUM_PARALLEL_PAGES = 15
+
+    # The amount of time to wait for pages to finish loading.
+    self._PAGE_LOAD_TIMEOUT_IN_SECONDS = 10
+
+    # The amount of time to wait for the retrieval of the URL of a tab.
+    self._TAB_URL_RETRIEVAL_TIMEOUT_IN_SECONDS = 1
+
+    # The amount of time to wait for a navigation to be committed.
+    self._NAVIGATION_COMMIT_WAIT_IN_SECONDS = 0.1
+
+    # A list of tuples (tab, initial_url). A navigation command has been sent
+    # to |tab|. |tab| had a URL of |initial_url| before the command was sent.
+    self._queued_tabs = []

[nednguyen, 2015/02/09 21:35:02]

What about making this a map from tabs to intial_urls?

[erikchen, 2015/02/10 00:45:02]

That loses the ordering, which matters for performance. OrderedDict does what
you want, but that requires python 2.7, and when I asked you, you couldn't
remember the oldest python version that telemetry supports.

[nednguyen, 2015/02/10 18:48:43]

Can you elaborate on why the ordering matters for performance? 

[erikchen, 2015/02/10 21:30:11]

Sending commands through the telemetry/Chromium bindings takes time
(BINDING_TIME). Let's say I'm simultaneously loading 10 tabs t1 through t10. I
send them commands sequentially, so t1 is likely to finish loading first, and
t10 likely to finish loading last.

Let's say that I wait for the tabs to load in order t10 through t1. t10 takes
the longest to load (let's say 5 seconds). Then I still need to check that t9
through t1 are loaded.

Total time = 5s + 9 * (BINDING_TIME). 

If instead I check t1 through t10, it's likely that t1 through t9 will finish
loading before t10 finishes. So

Total time = 5s.

All of this math assumes that t1 finishes loading first, and t10 finishes
loading last, which will only be true in aggregate on average, not all of the
time. 

There's no reason to turn an ordered list into an unordered dictionary,
especially if there is a small performance penalty. 

[nednguyen, 2015/02/10 22:11:37]

I want to see real benchmark to justify whether this optimization matters. It's
making the code more complicated. "Premature optimization is the root of all
evil"

[erikchen, 2015/02/11 02:45:50]

There is no optimization here. I'm using a list of tuples. You want me to use an
unordered dictionary. Using an unordered dictionary saves literally 0 lines of
code, and has the cost of losing the existing ordering. There's no benefit to
doing this. Don't quote aphorisms out of context.

[nednguyen, 2015/02/12 02:10:53]

Hmh, there was context, but the context has changed. You was using
_navigation_url_index to keep track of starting index in the queued list, which
make the implementation hard to follow. I asked you to use the map so you can
iterate through browser's tabs and use the map for getting tab's initial URL.
Anyway, with the most updated patch, the queued list implementation is as good
as map implementation.

[erikchen, 2015/02/12 03:02:58]

Your statement "You was using _navigation_url_index to keep track of starting
index in the queued list, which make the implementation hard to follow" is not
accurate. If you look at this patch set, you'll see that _navigation_url_index
is never indexed into _queued_tabs. Nothing is indexed into _queued_tabs.

The three operations performed on _queued_tabs are:
  - append
  - iterate
  - clear

Replacing _queued_tabs with a dictionary does not change any functionality or
clarity, just the container.

+
+    # The index of the first url that has not yet been navigated to.
+    self._navigation_url_index = 0

[nednguyen, 2015/02/09 21:35:02]

I don't like having this counter as a class's member. The more side-effect-free
methods, the easier it's to understand & unittest your implementation, and it's
less error prone.

[erikchen, 2015/02/10 00:45:02]

I've removed this member.

+
+  def Run(self, finder_options):
+    """
+    |finder_options| contains the directory of the input profile, the directory
+    to place the output profile, and sufficient information to choose a specific
+    browser binary.
+    """
+    try:
+      self._BrowserSetup(finder_options)

[nednguyen, 2015/02/09 21:35:02]

s/_BrowserSetup/_SetupBrowser

[erikchen, 2015/02/10 00:45:02]

I've renamed _BrowserSetup to _SetUpBrowser.

+      self._PerformNavigations()
+    except:

[nednguyen, 2015/02/09 21:35:02]

you don't need the "except:\nraise:"

[erikchen, 2015/02/10 00:45:02]

Done.

+      raise
+    finally:
+      self._BrowserTeardown()

[nednguyen, 2015/02/09 21:35:02]

s/_BrowserTeardown/_TeardownBrowser()

[erikchen, 2015/02/10 00:45:02]

I've renamed _BrowserTeardown to _TearDownBrowser.

+
+  def NavigationUrls(self):
+    """
+    Intended for subclass override. Returns a list of urls to be navigated to.
+    """
+    raise NotImplementedError()
+
+
+  def _GetPossibleBrowser(self, finder_options):
+    """Return a possible_browser with the given options."""
+    possible_browser = browser_finder.FindBrowser(finder_options)
+    if not possible_browser:
+      raise browser_finder_exceptions.BrowserFinderException(
+          'No browser found.\n\nAvailable browsers:\n%s\n' %
+          '\n'.join(browser_finder.GetAllAvailableBrowserTypes(finder_options)))
+    finder_options.browser_options.browser_type = (
+        possible_browser.browser_type)
+
+    return possible_browser
+
+  def _RetrieveTabUrl(self, tab):
+    """Retrives the URL of the tab."""
+    try:
+      return tab.EvaluateJavaScript('document.URL',
+          self._TAB_URL_RETRIEVAL_TIMEOUT_IN_SECONDS)
+    except exceptions.DevtoolsTargetCrashException:
+      return None
+
+  def _BatchNavigateTabs(self):
+    """Performs a batch of tab navigations with minimal delay."""
+    max_index = min(self._navigation_url_index + self._NUM_PARALLEL_PAGES,
+        len(self._navigation_urls))
+    timeout_in_seconds = 0
+
+    for i in range(self._navigation_url_index, max_index):
+      url = self._navigation_urls[i]
+      tab = self._browser.tabs[i % self._NUM_TABS]
+      initial_url = self._RetrieveTabUrl(tab)
+
+      try:
+        tab.Navigate(url, None, timeout_in_seconds)
+      except exceptions.DevtoolsTargetCrashException:
+        # We expect a time out, and don't mind if the webpage crashes. Ignore
+        # both exceptions.
+        pass
+
+      self._queued_tabs.append((tab, initial_url))
+    self._navigation_url_index = max_index
+
+  def _WaitForQueuedTabsToLoad(self):
+    """Waits for all the batch navigated tabs to finish loading."""
+    end_time = time.time() + self._PAGE_LOAD_TIMEOUT_IN_SECONDS
+    for tab, initial_url in self._queued_tabs:
+      seconds_to_wait = end_time - time.time()
+      seconds_to_wait = max(0, seconds_to_wait)
+
+      if seconds_to_wait == 0:
+        break
+
+      # Since we don't wait any time for the tab url navigation to commit, it's
+      # possible that the tab hasn't started navigating yet.
+      current_url = self._RetrieveTabUrl(tab)
+
+      if current_url == initial_url:
+        # If the navigation hasn't been committed yet, wait a small amount of
+        # time. Don't bother rechecking the condition, since it's also possible
+        # that the web page isn't processing javascript.
+        time.sleep(self._NAVIGATION_COMMIT_WAIT_IN_SECONDS)
+
+      try:
+        tab.WaitForDocumentReadyStateToBeComplete(seconds_to_wait)
+      except (util.TimeoutException, exceptions.DevtoolsTargetCrashException):
+        # Ignore time outs and web page crashes.
+        pass
+    self._queued_tabs = []
+
+  def _BrowserSetup(self, finder_options):
+    """
+    Finds the browser, starts the browser, and opens the requisite number of
+    tabs.
+    """
+    possible_browser = self._GetPossibleBrowser(finder_options)
+    self._browser = possible_browser.Create(finder_options)
+
+    for _ in range(self._NUM_TABS):
+      self._browser.tabs.New()
+
+  def _PerformNavigations(self):
+    """
+    Performs the navigations specified by |_navigation_urls| in large batches.
+    """
+    while True:
+      self._BatchNavigateTabs()
+      self._WaitForQueuedTabsToLoad()
+
+      if self._navigation_url_index == len(self._navigation_urls):
+        break
+
+  def _BrowserTeardown(self):
+    """
+    Teardown that is guaranteed to be executed before the instance is destroyed.
+    """
+    if self._browser:
+      self._browser.Close()
+      self._browser = None