Remove git lockfile flakiness on win (bot_update)

recipe_modules/bot_update/resources/bot_update.py

 #!/usr/bin/env python
 # Copyright 2014 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 # TODO(hinoka): Use logging.
 
 import cStringIO
 import codecs
 import copy
@@ skipping 188 matching lines @@
       print
       return outval
     if result is FAIL:
       break
     if result is RETRY and attempt < tries:
       sleep_backoff = 4 ** attempt
       sleep_time = random.randint(sleep_backoff, int(sleep_backoff * 1.2))
       print '===backing off, sleeping for %d secs===' % sleep_time
       time.sleep(sleep_time)
 
+      maybe_remove_win_lockfile(sys.platform, outval)

[agable, 2016/10/05 16:28:21]

If possible, this shouldn't be at the end of call(). The call function is
potentially used for many things, not just syncing git checkouts, and removing
the lockfile only makes sense in a subset of those cases.

[katthomas, 2016/10/06 21:50:20]

Done.

+
   raise SubprocessFailed('%s failed with code %d in %s after %d attempts.' %
                          (' '.join(args), code, cwd, attempt),
                          code, outval)
 
+# Windows sometimes has trouble deleting files. This can make git commands

[agable, 2016/10/05 16:28:20]

Use a docstring on line 218 instead of a comment above the definition.

[katthomas, 2016/10/06 21:50:20]

Done.

+# fhat rely on locks fail.
+def maybe_remove_win_lockfile(platform, error, retries=3):
+  if platform.startswith('win'):
+    match = re.search("(Unable to create ')(.*)(\.lock)(': File exists)", error)
+    if match:
+      filename = match.group(2) + '.lock'
+      print '===Attempting to remove lockfile %s===' % filename
+      for _ in xrange(retries):
+        exitcode = subprocess.call(['cmd.exe', '/c', 'del',

[Paweł Hajdan Jr., 2016/10/03 11:53:22]

Please consider breaking the locks only once, at the beginning of
bot_update/gclient invocation.

gclient could recursively do so for all repos it manages. We already do that for
the git cache.

Also consider killing possibly stuck git processes in
build\scripts\slave\kill_processes.py .

[agable, 2016/10/05 16:28:21]

Agreed with this comment -- it makes more sense to me to aggressively break
locks at the beginning, rather than reactively doing it after things fail.

And also yes it's a good tip to look into kill_processes.py as well, but I'd
only do that if this is unsuccessful.

Finally, if you do move the call to this out from call(), then you should use
call() instead of subprocess.call().

[katthomas, 2016/10/06 21:50:20]

Done. I put the lockbreaking before running any git command, with the logic that
if we do it once at the beginning, we could end up with the same problem if
subsequent git calls leave behind a lockfile. Thoughts?

+                                    '/f', '/q', os.path.normcase(filename)])
+        if exitcode == 0:
+          return
+        time.sleep(3)
+      print '===Failed to remove lockfile %s====' % filename
 
 def git(*args, **kwargs):  # pragma: no cover
   """Wrapper around call specifically for Git commands."""
   if args and args[0] == 'cache':
     # Rewrite "git cache" calls into "python git_cache.py".
     cmd = (sys.executable, '-u', GIT_CACHE_PATH) + args[1:]
   else:
     git_executable = 'git'
     # On windows, subprocess doesn't fuzzy-match 'git' to 'git.bat', so we
     # have to do it explicitly. This is better than passing shell=True.
@@ skipping 867 matching lines @@
     # download patch failure is still an infra problem.
     if e.code == 3:
       # Patch download problem.
       return 87
     # Genuine patch problem.
     return 88
 
 
 if __name__ == '__main__':
   sys.exit(main())