[run_isolated] Tolerate cache uninstall errors.

client/run_isolated.py

 #!/usr/bin/env python
 # Copyright 2012 The LUCI Authors. All rights reserved.
 # Use of this source code is governed under the Apache License, Version 2.0
 # that can be found in the LICENSE file.
 
 """Runs a command with optional isolated input/output.
 
 Despite name "run_isolated", can run a generic non-isolated command specified as
 args.
 
@@ skipping 1035 matching lines @@
     caches = [
       (os.path.join(run_dir, unicode(relpath)), name)
       for name, relpath in options.named_caches
     ]
     with named_cache_manager.open():
       for path, name in caches:
         named_cache_manager.install(path, name)
     try:
       yield
     finally:
+      # Uninstall each named cache, returning it to the cache pool. If an
+      # uninstall fails for a given cache, it will remain in the task's
+      # temporary space, get cleaned up by the Swarming bot, and be lost.
+      #
+      # If the Swarming bot cannot clean up the cache, it will handle it like
+      # any other bot file that could not be removed.
       with named_cache_manager.open():
         for path, name in caches:
-          named_cache_manager.uninstall(path, name)
+          try:
+            named_cache_manager.uninstall(path, name)

[Vadim Sh., 2017/07/07 18:01:46]

does this thing leave 'named_cache_manager' in consistent state when an
exception happens?

[dnj, 2017/07/07 18:14:12]

Good question. It seems to:
https://cs.chromium.org/chromium/infra/luci/client/named_cache.py?q=named_cac...

If the "rename" succeeds but the LRU isn't updated, a cache entry won't be
present in the LRU. Next time the cache is referenced, it will be re-used and
added to the LRU next time (assuming it succeeds). "trim" also is resilient to
directories that aren't in LRU, as it does a "listdir".

All of the other operations are filesystem state, so nothing new is introduced
here.

+          except named_cache.Error:
+            logging.exception('Error while removing named cache %r at %r. '
+                              'The cache will be lost.', path, name)
 
   try:
     if options.isolate_server:
       storage = isolateserver.get_storage(
           options.isolate_server, options.namespace)
       with storage:
         # Hashing schemes used by |storage| and |isolate_cache| MUST match.
         assert storage.hash_algo == isolate_cache.hash_algo
         return run_tha_test(
             args,
@@ skipping 29 matching lines @@
     return 1
 
 
 if __name__ == '__main__':
   subprocess42.inhibit_os_error_reporting()
   # Ensure that we are always running with the correct encoding.
   fix_encoding.fix_encoding()
   file_path.enable_symlink()
 
   sys.exit(main(sys.argv[1:]))