Only schedule termination task if an unconnected MP bot is dead

Only schedule termination task if an unconnected MP bot is dead

Previously this was happening no matter what, meaning we were hardly
giving any chance to MP bots to actually connect.

Bug: 717769
Review-Url: https://codereview.chromium.org/2951023002
Committed: https://github.com/luci/luci-py/commit/3fa4c520dddd82ed190152709e0a54b35faa3bae

[smut, 2017-06-21 01:20:33]

smut@google.com changed reviewers:
+ vadimsh@chromium.org

[smut, 2017-06-21 01:20:34]

I'm surprised nobody noticed/complained. The intention is for MP bots which
don't connect within 10 minutes to be terminated and released.

As implemented if the MP bot didn't connect immediately it would be terminated,
and then after 10 minutes it would be released. Since the cron frequency is
every minute, this means that if a bot doesn't connect within 60 seconds it gets
terminated.

[smut, 2017-06-21 01:21:43]

Patchset #1 (id:1) has been deleted

[smut, 2017-06-21 01:23:38]

Patchset #1 (id:20001) has been deleted

[Vadim Sh., 2017-06-21 01:24:30]

lol

lgmt

[Vadim Sh., 2017-06-21 01:24:38]

On 2017/06/21 01:24:30, Vadim Sh. wrote:
> lol
> 
> lgmt

err lgtm

[smut, 2017-06-21 01:25:22]

The CQ bit was checked by smut@google.com

[commit-bot: I haz the power, 2017-06-21 01:25:33]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Vadim Sh., 2017-06-21 01:26:02]

Why we didn't notice? Does MP gives released bots right back to Swarming or it
actually terminates them?

[smut, 2017-06-21 01:29:46]

On 2017/06/21 01:26:02, Vadim Sh. wrote:
> Why we didn't notice? Does MP gives released bots right back to Swarming or it
> actually terminates them?

It actually deletes them, so you have to wait for GCE Backend to spin up a new
VM and there is definitely a delay. The VMs that arrived within 60 seconds would
work as normal. Maybe most VMs arrive very quickly (I think it's unlikely) or we
have far too much capacity allocated.

[commit-bot: I haz the power, 2017-06-21 01:30:03]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1498008322361020,
"parent_rev": "6551b9f9aa7299794f98be64bd53cd91dbbedc2e", "commit_rev":
"3fa4c520dddd82ed190152709e0a54b35faa3bae"}

[commit-bot: I haz the power, 2017-06-21 01:30:08]

Description was changed from

==========
Only schedule termination task if an unconnected MP bot is dead

Previously this was happening no matter what, meaning we were hardly
giving any chance to MP bots to actually connect.

Bug: 717769
==========

to

==========
Only schedule termination task if an unconnected MP bot is dead

Previously this was happening no matter what, meaning we were hardly
giving any chance to MP bots to actually connect.

Bug: 717769
Review-Url: https://codereview.chromium.org/2951023002
Committed:
https://github.com/luci/luci-py/commit/3fa4c520dddd82ed190152709e0a54b35faa3bae
==========

[commit-bot: I haz the power, 2017-06-21 01:30:09]

Committed patchset #1 (id:40001) as
https://github.com/luci/luci-py/commit/3fa4c520dddd82ed190152709e0a54b35faa3bae

[smut, 2017-06-21 01:31:24]

On 2017/06/21 01:29:46, smut wrote:
> On 2017/06/21 01:26:02, Vadim Sh. wrote:
> > Why we didn't notice? Does MP gives released bots right back to Swarming or
it
> > actually terminates them?
> 
> It actually deletes them, so you have to wait for GCE Backend to spin up a new
> VM and there is definitely a delay. The VMs that arrived within 60 seconds
would
> work as normal. Maybe most VMs arrive very quickly (I think it's unlikely) or
we
> have far too much capacity allocated.

Actually it might be possible that most or enough VMs arrive within 60 seconds.
The 60 second timer only starts once MP says the VM is yours, which only happens
once it has been booted. The only thing that would cause a delay is if startup
scripts take an exceptionally long time to complete.