Add HttpsMonitor

infra_libs/httplib2_utils.py

Index: infra_libs/httplib2_utils.py
diff --git a/infra_libs/httplib2_utils.py b/infra_libs/httplib2_utils.py
index fcb9fe387f5c52b8bf5a64eb6b4e578b229990dc..8b0e16f2d1b5ec70d8843ca072470229b9f3e3a2 100644
--- a/infra_libs/httplib2_utils.py
+++ b/infra_libs/httplib2_utils.py
@@ -15,6 +15,7 @@ import time
 import httplib2
 import oauth2client.client
 
+from googleapiclient import errors
 from infra_libs.ts_mon.common import http_metrics
 
 DEFAULT_SCOPES = ['email']
@@ -161,6 +162,44 @@ def get_authenticated_http(credentials_filename,
     http = httplib2.Http(timeout=timeout)
   return creds.authorize(http)
 
+class RetryableHttp(object):

[Sergey Berezin, 2016/07/06 20:13:28]

nit (spelling): Retriable

[Sergey Berezin, 2016/07/06 20:13:29]

Inherit from httplib2.Http, and call base class's __init__().

[tnn, 2016/07/06 22:05:18]

Done.

+  """A httplib2.Http object that retries on failure."""
+
+  def __init__(self, http_obj, max_tries=5,
+               retrying_statuses=range(500,599)):

[Sergey Berezin, 2016/07/06 20:13:28]

For efficiency, I'd either make it a set(range(500, 599)), or a function. In
fact, I'd prefer a function.

Also, you might want to default to None, and assign the actual default as

  self.retrying_status = retrying_statuses or <default_value>

It will keep pylint happier, and is friendlier to **kwargs pattern.

[tnn, 2016/07/06 22:05:17]

Done.

+    """
+    Args:
+      http_obj: an httplib2.Http instance
+      max_tries: a number of maximum tries
+      retrying_status_codes: a set of HTTP status codes that cause a retry
+    """
+    self.http_obj = http_obj
+    self.max_tries = max_tries
+    self.retrying_statuses = retrying_statuses
+
+  def request(self, uri, method="GET", body=None, *args, **kwargs):

[Sergey Berezin, 2016/07/06 20:13:28]

nit: use single quotes for strings if possible: 'GET'.

(the InstrumentedHttp.request is using "wrong" quotes...)

[tnn, 2016/07/06 22:05:18]

Done.

+    for i in range(1, self.max_tries + 1):
+      try:
+        response, content = self.http_obj.request(
+	    uri, method, body, *args, **kwargs)

[Sergey Berezin, 2016/07/06 20:13:29]

nit: indent 4 spaces relative to the prev. line.

[tnn, 2016/07/06 22:05:18]

I swear it looked like 4 spaces in my vim :S It turned out there was one tab
character in here.

Done.

+
+        if response.status in self.retrying_statuses:
+          logging.info("RetryableHttp retries (%d) on receiving status: %d",

[Sergey Berezin, 2016/07/06 20:13:28]

nit (wording suggestion):
  logging.info(
      'RetriableHttp: attempt %d received status %d, %s',
      i, response.status,
      'final attempt' if i == self._max_tries else 'will retry')

[tnn, 2016/07/06 22:05:17]

Done.

+                       i, response.status)
+          continue
+      except (ValueError, errors.Error,
+	      socket.timeout, socket.error, socket.herror, socket.gaierror,

[Sergey Berezin, 2016/07/06 20:13:28]

nit: we always indent arguments to align:

  your_method(
      arg1, arg2,  # note: 4-space indent, unlike 2-space for statements
      arg3, arg4)

OR:

  your_method(arg1, arg2,
              arg3, arg4)

[tnn, 2016/07/06 22:05:18]

Done.

+	      httplib2.HttpLib2Error) as error:
+        if i == self.max_tries:
+          logging.warning("RetryableHttp fails on exception: %s", error)

[Sergey Berezin, 2016/07/06 20:13:29]

nit: 'single quotes'

[tnn, 2016/07/06 22:05:18]

Done.

+          raise
+        else:
+          logging.info("RetryableHttp retries (%d) on exception: %s", i, error)

[Sergey Berezin, 2016/07/06 20:13:28]

nit: 'single quotes', and similar wording suggestion as above.

[tnn, 2016/07/06 22:05:18]

Done.

+          continue
+      break
+
+    return response, content
+
 
 class InstrumentedHttp(httplib2.Http):
   """A httplib2.Http object that reports ts_mon metrics about its requests."""