Complete CIPD integration

Complete CIPD integration

These are last bits of CIPD integration MVP:

- pipe cipd parameters from bot manigest to run_isolated.py
- pipe cipd stats from run_isolated.py back to server and store it in
  datastore
- update user task page to display the list of CIPD input
  and installation duration

R=maruel@chromium.org
BUG=601022
Committed: https://github.com/luci/luci-py/commit/006f9a416983c09c8bc188358ee6aae7ebe6e1cb

[nodir, 2016-06-06 17:20:44]

The CQ bit was checked by nodir@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-06-06 17:20:46]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/2043853002/20001

[nodir, 2016-06-06 17:21:03]

PTAL. Will add tests

[commit-bot: I haz the power, 2016-06-06 17:23:56]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-06-06 17:23:57]

Dry run: This issue passed the CQ dry run.

[M-A Ruel, 2016-06-06 23:54:28]

This CL looks good, will finalise review once the dependent one is done.

https://codereview.chromium.org/2043853002/diff/20001/appengine/swarming/swar...
File appengine/swarming/swarming_bot/bot_code/task_runner.py (right):

https://codereview.chromium.org/2043853002/diff/20001/appengine/swarming/swar...
appengine/swarming/swarming_bot/bot_code/task_runner.py:108:
'--cipd-client-package', to_pkg(
I'd prefer the to_pkg( on the next line then the function call wouldn't be
split.

[nodir, 2016-06-09 23:36:28]

https://codereview.chromium.org/2043853002/diff/20001/appengine/swarming/swar...
File appengine/swarming/swarming_bot/bot_code/task_runner.py (right):

https://codereview.chromium.org/2043853002/diff/20001/appengine/swarming/swar...
appengine/swarming/swarming_bot/bot_code/task_runner.py:108:
'--cipd-client-package', to_pkg(
On 2016/06/06 23:54:28, M-A Ruel wrote:
> I'd prefer the to_pkg( on the next line then the function call wouldn't be
> split.

Done.

[M-A Ruel, 2016-06-10 02:06:42]

lgtm

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/hand...
File appengine/swarming/handlers_bot.py (right):

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/hand...
appengine/swarming/handlers_bot.py:606: duration=cipd_stats.get('duration'))
and client?

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/swar...
File appengine/swarming/swarming_bot/bot_code/task_runner.py (right):

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/swar...
appengine/swarming/swarming_bot/bot_code/task_runner.py:116:
cmd.extend(['--cipd-package', to_pkg(p)])
(FYI only) Eventually we may have to pass this information through a file as we
may blow the 8kb limit (Thanks Windows). Should be fine initially but this could
blow up as people starts adding more packages with long URLs or whatnot. We
should be fine for a while.

[nodir, 2016-06-10 15:46:32]

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/hand...
File appengine/swarming/handlers_bot.py (right):

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/hand...
appengine/swarming/handlers_bot.py:606: duration=cipd_stats.get('duration'))
On 2016/06/10 02:06:42, M-A Ruel wrote:
> and client?

so OperationStats does not have a place to put get_client_duration. The only
option to keep it in datastore I see is to make a new model, but I am not sure
it is worth the effort.

[nodir, 2016-06-10 16:10:01]

The CQ bit was checked by nodir@chromium.org

[commit-bot: I haz the power, 2016-06-10 16:10:10]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/2043853002/40001

[commit-bot: I haz the power, 2016-06-10 16:12:44]

Description was changed from

==========
Complete CIPD integration

These are last bits of CIPD integration MVP:

- pipe cipd parameters from bot manigest to run_isolated.py
- pipe cipd stats from run_isolated.py back to server and store it in
  datastore
- update user task page to display the list of CIPD input
  and installation duration

R=maruel@chromium.org
BUG=601022
==========

to

==========
Complete CIPD integration

These are last bits of CIPD integration MVP:

- pipe cipd parameters from bot manigest to run_isolated.py
- pipe cipd stats from run_isolated.py back to server and store it in
  datastore
- update user task page to display the list of CIPD input
  and installation duration

R=maruel@chromium.org
BUG=601022

Committed:
https://github.com/luci/luci-py/commit/006f9a416983c09c8bc188358ee6aae7ebe6e1cb
==========

[commit-bot: I haz the power, 2016-06-10 16:12:45]

Committed patchset #3 (id:40001) as
https://github.com/luci/luci-py/commit/006f9a416983c09c8bc188358ee6aae7ebe6e1cb

[M-A Ruel, 2016-06-17 17:38:32]

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/swar...
File appengine/swarming/swarming_bot/bot_code/task_runner.py (right):

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/swar...
appengine/swarming/swarming_bot/bot_code/task_runner.py:122: '--cache',
os.path.join(bot_dir, 'isolated_cache'),
Oh, this is problematic, now all the bots have a stale 'cache' directory and
this is getting filled up. It's kind of too late already as this change was
deployed.

[nodir, 2016-06-17 18:58:57]

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/swar...
File appengine/swarming/swarming_bot/bot_code/task_runner.py (right):

https://codereview.chromium.org/2043853002/diff/40001/appengine/swarming/swar...
appengine/swarming/swarming_bot/bot_code/task_runner.py:122: '--cache',
os.path.join(bot_dir, 'isolated_cache'),
On 2016/06/17 17:38:32, M-A Ruel wrote:
> Oh, this is problematic, now all the bots have a stale 'cache' directory and
> this is getting filled up. It's kind of too late already as this change was
> deployed.

I should not have done that