[run_isolated] Tolerate cache uninstall errors.

[run_isolated] Tolerate cache uninstall errors.

If a named cache cannot be uninstalled, the Swarming bot will fail with
an unfriendly code path and the task will terminate as BOT_DIED. This
can happen if a zombie process lingers from a task and retains a handle to
the named cache.

Swarming already has code paths to handle zombie processes and task
space purge errors. This patch makes it so that named cache deletion
failures fall through to standard cleanup code instead of raising an
exception.

BUG=chromium:740109
TEST=None
R=maruel@chromium.org, vadimsh@chromium.org

Review-Url: https://codereview.chromium.org/2973113003
Committed: https://github.com/luci/luci-py/commit/bd3cbc5ca8f6fd82345bb9073364c56bf2b73130

[dnj, 2017-07-07 16:00:06]

dnj@chromium.org changed reviewers:
+ dnj@chromium.org, iannucci@chromium.org

[dnj, 2017-07-07 16:00:07]

+iannucci@

[dnj, 2017-07-07 16:00:41]

The CQ bit was checked by dnj@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-07-07 16:00:52]

Dry run: CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-07-07 16:03:21]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-07-07 16:03:22]

Dry run: This issue passed the CQ dry run.

[Vadim Sh., 2017-07-07 18:01:47]

lgtm with question

https://codereview.chromium.org/2973113003/diff/1/client/run_isolated.py
File client/run_isolated.py (right):

https://codereview.chromium.org/2973113003/diff/1/client/run_isolated.py#newc...
client/run_isolated.py:1065: named_cache_manager.uninstall(path, name)
does this thing leave 'named_cache_manager' in consistent state when an
exception happens?

[dnj, 2017-07-07 18:14:11]

The CQ bit was checked by dnj@chromium.org

[dnj, 2017-07-07 18:14:12]

lgtm

https://codereview.chromium.org/2973113003/diff/1/client/run_isolated.py
File client/run_isolated.py (right):

https://codereview.chromium.org/2973113003/diff/1/client/run_isolated.py#newc...
client/run_isolated.py:1065: named_cache_manager.uninstall(path, name)
On 2017/07/07 18:01:46, Vadim Sh. wrote:
> does this thing leave 'named_cache_manager' in consistent state when an
> exception happens?

Good question. It seems to:
https://cs.chromium.org/chromium/infra/luci/client/named_cache.py?q=named_cac...

If the "rename" succeeds but the LRU isn't updated, a cache entry won't be
present in the LRU. Next time the cache is referenced, it will be re-used and
added to the LRU next time (assuming it succeeds). "trim" also is resilient to
directories that aren't in LRU, as it does a "listdir".

All of the other operations are filesystem state, so nothing new is introduced
here.

[commit-bot: I haz the power, 2017-07-07 18:14:22]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-07-07 18:16:44]

CQ is committing da patch.

Bot data: {"patchset_id": 1, "attempt_start_ts": 1499451251803110, "parent_rev":
"32c3d35dd963a5798200fbbdf18f9025317ec5e8", "commit_rev":
"bd3cbc5ca8f6fd82345bb9073364c56bf2b73130"}

[commit-bot: I haz the power, 2017-07-07 18:16:47]

Description was changed from

==========
[run_isolated] Tolerate cache uninstall errors.

If a named cache cannot be uninstalled, the Swarming bot will fail with
an unfriendly code path and the task will terminate as BOT_DIED. This
can happen if a zombie process lingers from a task and retains a handle to
the named cache.

Swarming already has code paths to handle zombie processes and task
space purge errors. This patch makes it so that named cache deletion
failures fall through to standard cleanup code instead of raising an
exception.

BUG=chromium:740109
TEST=None
R=maruel@chromium.org, vadimsh@chromium.org
==========

to

==========
[run_isolated] Tolerate cache uninstall errors.

If a named cache cannot be uninstalled, the Swarming bot will fail with
an unfriendly code path and the task will terminate as BOT_DIED. This
can happen if a zombie process lingers from a task and retains a handle to
the named cache.

Swarming already has code paths to handle zombie processes and task
space purge errors. This patch makes it so that named cache deletion
failures fall through to standard cleanup code instead of raising an
exception.

BUG=chromium:740109
TEST=None
R=maruel@chromium.org, vadimsh@chromium.org

Review-Url: https://codereview.chromium.org/2973113003
Committed:
https://github.com/luci/luci-py/commit/bd3cbc5ca8f6fd82345bb9073364c56bf2b73130
==========

[commit-bot: I haz the power, 2017-07-07 18:16:48]

Committed patchset #1 (id:1) as
https://github.com/luci/luci-py/commit/bd3cbc5ca8f6fd82345bb9073364c56bf2b73130