Add domain checking to returned HTTPResponses

tools/chrome_proxy/webdriver/common.py

Index: tools/chrome_proxy/webdriver/common.py
diff --git a/tools/chrome_proxy/webdriver/common.py b/tools/chrome_proxy/webdriver/common.py
index 05b2c89dc0e0bc453b78d2f0d2ce3cf131d8cc49..2ff3f0b865972fb82dd9f05e7fb0877d78538c32 100644
--- a/tools/chrome_proxy/webdriver/common.py
+++ b/tools/chrome_proxy/webdriver/common.py
@@ -13,6 +13,7 @@ import sys
 import time
 import traceback
 import unittest
+import urlparse
 
 sys.path.append(os.path.join(os.path.dirname(__file__), os.pardir, os.pardir,
   os.pardir, 'third_party', 'webdriver', 'pylib'))
@@ -357,7 +358,8 @@ class TestDriver:
       len(all_messages), method_filter)
     return all_messages
 
-  def GetHTTPResponses(self, include_favicon=False):
+  def GetHTTPResponses(self, include_favicon=False,
+      only_include_loaded_domain=True):

[sclittle, 2016/12/19 19:50:09]

I'm not sure this is the best way to solve the problem of data:// or about:blank
URLs being requested. Could you just filter out all non-http/https URLs by
default, or are there other URLs from other domains that pose problems?

[Robert Ogden, 2016/12/19 20:23:10]

Filtering on presence of the scheme (http,https) would work. The idea behind
domains is not a concrete one, just a thought/fear of unexpected behavior for
test sites that use cross-origin requests.

[sclittle, 2016/12/19 21:17:08]

Hmm - I think I see what you mean, but I'm worried that ignoring the
cross-origin requests could cause more problems than it solves. E.g., suppose
there's an integration test to check if bypass works. If the request that caused
the bypass gets ignored here because it was cross origin, it'll be harder to
debug.

Plus, hopefully it shouldn't be too hard to have our test pages only make
first-party requests. If they do make cross-origin requests, then that's
probably on purpose and we likely don't want to ignore those requests anyways.

WDYT?

[Robert Ogden, 2016/12/19 22:17:46]

SGTM. After looking through the pydoc, I ended up checking the presence of a
net_loc (or domain as I know it) instead of a scheme, since scheme='data' where
url='data:,,' which has been happening a lot recently.

     """Parses the Performance Logs and returns a list of HTTPResponse objects.
 
     Use caution when calling this function  multiple times. Only responses
@@ -366,6 +368,8 @@ class TestDriver:
 
     Args:
       include_favicon: A bool that if True will include responses for favicons.
+      only_include_loaded_domain: If True, only responses with the same domain
+        as the last requested page will be included in the result.
     Returns:
       A list of HTTPResponse objects, each representing a single completed HTTP
       transaction by Chrome.
@@ -392,8 +396,14 @@ class TestDriver:
       response = MakeHTTPResponse(message)
       self._logger.debug('New HTTPResponse: %s', str(response))
       is_favicon = response.url.endswith('favicon.ico')
-      if not is_favicon or include_favicon:
+      is_same_loaded_domain = (urlparse.urlparse(response.url).netloc ==
+        urlparse.urlparse(self._url).netloc)
+      if ((not is_favicon or include_favicon) and
+          (not only_include_loaded_domain or is_same_loaded_domain)):
         all_responses.append(response)
+      else:
+        self._logger.info("Skipping HTTPResponse with url=%s in returned logs.",
+          response.url)
     self._logger.info('%d new HTTPResponse objects found in the logs %s '
       'favicons', len(all_responses), ('including' if include_favicon else
       'not including'))