Remove git lockfile flakiness on win (bot_update)

recipe_modules/bot_update/resources/bot_update.py

Index: recipe_modules/bot_update/resources/bot_update.py
diff --git a/recipe_modules/bot_update/resources/bot_update.py b/recipe_modules/bot_update/resources/bot_update.py
index ca91dbdf98d15876634feeb65406ba9007608523..0dd780757c4a5ae6731693247e1bdc6f4def9aee 100755
--- a/recipe_modules/bot_update/resources/bot_update.py
+++ b/recipe_modules/bot_update/resources/bot_update.py
@@ -206,10 +206,27 @@ def call(*args, **kwargs):  # pragma: no cover
       print '===backing off, sleeping for %d secs===' % sleep_time
       time.sleep(sleep_time)
 
+      maybe_remove_win_lockfile(sys.platform, outval)

[agable, 2016/10/05 16:28:21]

If possible, this shouldn't be at the end of call(). The call function is
potentially used for many things, not just syncing git checkouts, and removing
the lockfile only makes sense in a subset of those cases.

[katthomas, 2016/10/06 21:50:20]

Done.

+
   raise SubprocessFailed('%s failed with code %d in %s after %d attempts.' %
                          (' '.join(args), code, cwd, attempt),
                          code, outval)
 
+# Windows sometimes has trouble deleting files. This can make git commands

[agable, 2016/10/05 16:28:20]

Use a docstring on line 218 instead of a comment above the definition.

[katthomas, 2016/10/06 21:50:20]

Done.

+# fhat rely on locks fail.
+def maybe_remove_win_lockfile(platform, error, retries=3):
+  if platform.startswith('win'):
+    match = re.search("(Unable to create ')(.*)(\.lock)(': File exists)", error)
+    if match:
+      filename = match.group(2) + '.lock'
+      print '===Attempting to remove lockfile %s===' % filename
+      for _ in xrange(retries):
+        exitcode = subprocess.call(['cmd.exe', '/c', 'del',

[Paweł Hajdan Jr., 2016/10/03 11:53:22]

Please consider breaking the locks only once, at the beginning of
bot_update/gclient invocation.

gclient could recursively do so for all repos it manages. We already do that for
the git cache.

Also consider killing possibly stuck git processes in
build\scripts\slave\kill_processes.py .

[agable, 2016/10/05 16:28:21]

Agreed with this comment -- it makes more sense to me to aggressively break
locks at the beginning, rather than reactively doing it after things fail.

And also yes it's a good tip to look into kill_processes.py as well, but I'd
only do that if this is unsuccessful.

Finally, if you do move the call to this out from call(), then you should use
call() instead of subprocess.call().

[katthomas, 2016/10/06 21:50:20]

Done. I put the lockbreaking before running any git command, with the logic that
if we do it once at the beginning, we could end up with the same problem if
subsequent git calls leave behind a lockfile. Thoughts?

+                                    '/f', '/q', os.path.normcase(filename)])
+        if exitcode == 0:
+          return
+        time.sleep(3)
+      print '===Failed to remove lockfile %s====' % filename
 
 def git(*args, **kwargs):  # pragma: no cover
   """Wrapper around call specifically for Git commands."""