Switch bot_update to use exponential backoff on clone

recipe_modules/bot_update/resources/bot_update.py

Index: recipe_modules/bot_update/resources/bot_update.py
diff --git a/recipe_modules/bot_update/resources/bot_update.py b/recipe_modules/bot_update/resources/bot_update.py
index 03b14e502fa4907593fe3051d7d64060791a1b38..b1c78681d744b15f7686716c7ecc49749a9d3cad 100755
--- a/recipe_modules/bot_update/resources/bot_update.py
+++ b/recipe_modules/bot_update/resources/bot_update.py
@@ -543,10 +543,11 @@ def git_checkout(solutions, revisions, shallow, refs, git_cache_dir):
         raise Exception('%s exists after cache unlock' % filename)
   first_solution = True
   for sln in solutions:
-    # This is so we can loop back and try again if we need to wait for the
-    # git mirrors to update from SVN.
+    # Just in case we're hitting a different git server than the one from
+    # which the target revision was polled, we retry some.
     done = False
-    tries_left = 60
+    deadline = time.time() + 5*60  # Two minutes worth of tries.

[iannucci, 2016/09/09 01:33:07]

<suspicious_fry/>

[agable, 2016/09/09 01:40:02]

lol whoops. Fixed, and changed to 1 minute because geronimooooo

+    tries = 0
     while not done:
       name = sln['name']
       url = sln['url']
@@ -582,14 +583,17 @@ def git_checkout(solutions, revisions, shallow, refs, git_cache_dir):
         done = True
       except SubprocessFailed as e:
         # Exited abnormally, theres probably something wrong.
-        # Lets wipe the checkout and try again.
-        tries_left -= 1
-        if tries_left > 0:
+        if time.time() < deadline:
+          tries += 1
+          sleep_secs = 2**tries
           print 'Something failed: %s.' % str(e)
-          print 'waiting 5 seconds and trying again...'
-          time.sleep(5)
+          print 'waiting %s seconds and trying again...' % sleep_secs
+          time.sleep(sleep_secs)
         else:
+          overrun = time.time() - deadline
+          print 'Ran %s seconds past deadline. Aborting.' % overrun
           raise
+        # Lets wipe the checkout and try again.
         remove(sln_dir)

[iannucci, 2016/09/09 01:33:07]

could this move up into the `if` clause? It clearly doesn't run in the else
case.

Actually what about
```
if <deadline_exceeded>:
  raise

# retry case
```

[agable, 2016/09/09 01:40:02]

Good call, done.

 
     git('clean', '-dff', cwd=sln_dir)