Add bots.count endpoint

Add bots.count endpoint

This is analogous to tasks.count, with the addition of a count of dead,
quarantined and busy bots.

BUG=631047
Committed: https://github.com/luci/luci-py/commit/42b792562b6445d315416c6e306d24a71fa3626d

[M-A Ruel, 2016-08-01 15:33:13]

maruel@chromium.org changed reviewers:
+ maruel@chromium.org

[M-A Ruel, 2016-08-01 15:33:14]

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
File appengine/swarming/handlers_endpoints.py (right):

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:647: q =
bot_management.BotInfo.query().order(bot_management.BotInfo.key)
order is not needed for counting.

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:649: if not ':' in d:
this check is not needed because you already check for this at line 652.

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:650: raise
endpoints.BadRequestException('Invalid dimensions')
...: %s' % d)
?

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:655: bots, cursor =
datastore_utils.fetch_page(q)
you don't need to fetch a page, just count, see line 410.

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/swar...
File appengine/swarming/swarming_rpcs.py (right):

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/swar...
appengine/swarming/swarming_rpcs.py:375: 
2 lines

[kjlubick, 2016-08-01 17:25:12]

Description was changed from

==========
Adding bots count

BUG=
==========

to

==========
Add bots.count endpoint

This is analogous to tasks.count, with the addition of a count of dead,
quarantined and busy bots. 

BUG=631047
==========

[kjlubick, 2016-08-01 17:25:52]

Description was changed from

==========
Add bots.count endpoint

This is analogous to tasks.count, with the addition of a count of dead,
quarantined and busy bots. 

BUG=631047
==========

to

==========
Add bots.count endpoint

This is analogous to tasks.count, with the addition of a count of dead,
quarantined and busy bots.

BUG=631047
==========

[kjlubick, 2016-08-01 17:26:32]

Tests have been added, PTAL

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
File appengine/swarming/handlers_endpoints.py (right):

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:647: q =
bot_management.BotInfo.query().order(bot_management.BotInfo.key)
On 2016/08/01 at 15:33:14, M-A Ruel wrote:
> order is not needed for counting.

Done.

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:649: if not ':' in d:
On 2016/08/01 at 15:33:14, M-A Ruel wrote:
> this check is not needed because you already check for this at line 652.

Done.

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:650: raise
endpoints.BadRequestException('Invalid dimensions')
On 2016/08/01 at 15:33:14, M-A Ruel wrote:
> ...: %s' % d)
> ?

Done.

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/hand...
appengine/swarming/handlers_endpoints.py:655: bots, cursor =
datastore_utils.fetch_page(q)
On 2016/08/01 at 15:33:14, M-A Ruel wrote:
> you don't need to fetch a page, just count, see line 410.

Done.

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/swar...
File appengine/swarming/swarming_rpcs.py (right):

https://codereview.chromium.org/2198063002/diff/20001/appengine/swarming/swar...
appengine/swarming/swarming_rpcs.py:375: 
On 2016/08/01 at 15:33:14, M-A Ruel wrote:
> 2 lines

Done.

[M-A Ruel, 2016-08-01 18:27:19]

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/han...
File appengine/swarming/handlers_endpoints.py (right):

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/han...
appengine/swarming/handlers_endpoints.py:656: dead =
q.filter(bot_management.BotInfo.last_seen_ts < timeout).count()
you are serializing 3 queries that can be perfectly parallelized.

Starting at line 653, create q_dead and q_quarantined. Then call count_async()
on the 3 queries and get_result() afterward so that the 3 queries run in
parallel.

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/swa...
File appengine/swarming/swarming_rpcs.py (right):

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_rpcs.py:380: now = message_types.DateTimeField(2)
make it the first, otherwise it's kind of odd order.

[kjlubick, 2016-08-01 19:15:20]

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/han...
File appengine/swarming/handlers_endpoints.py (right):

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/han...
appengine/swarming/handlers_endpoints.py:656: dead =
q.filter(bot_management.BotInfo.last_seen_ts < timeout).count()
On 2016/08/01 at 18:27:19, M-A Ruel wrote:
> you are serializing 3 queries that can be perfectly parallelized.
> 
> Starting at line 653, create q_dead and q_quarantined. Then call count_async()
on the 3 queries and get_result() afterward so that the 3 queries run in
parallel.

Good call.  Done.

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/swa...
File appengine/swarming/swarming_rpcs.py (right):

https://codereview.chromium.org/2198063002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_rpcs.py:380: now = message_types.DateTimeField(2)
On 2016/08/01 at 18:27:19, M-A Ruel wrote:
> make it the first, otherwise it's kind of odd order.

Done.

[M-A Ruel, 2016-08-01 19:18:37]

lgtm with two fixes, thanks for driving this through!

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
File appengine/swarming/handlers_endpoints.py (right):

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
appengine/swarming/handlers_endpoints.py:656: q_dead =
(q.filter(bot_management.BotInfo.last_seen_ts < timeout)
optional style nit: I'd prefer:
q_dead = q.filter(
    bot_management.BotInfo.last_seen_ts < timeout).count_async()

same for the next one.

technically, they are not queries but ndb.Future, so using f_ instead of q_
would make more sense.

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
File appengine/swarming/handlers_endpoints_test.py (right):

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
appengine/swarming/handlers_endpoints_test.py:1447: u'count': unicode(3),
Use actual values, it's shorter, e.g. u'3'

[kjlubick, 2016-08-01 19:25:43]

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
File appengine/swarming/handlers_endpoints.py (right):

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
appengine/swarming/handlers_endpoints.py:656: q_dead =
(q.filter(bot_management.BotInfo.last_seen_ts < timeout)
On 2016/08/01 at 19:18:37, M-A Ruel wrote:
> optional style nit: I'd prefer:
> q_dead = q.filter(
>     bot_management.BotInfo.last_seen_ts < timeout).count_async()
> 
> same for the next one.
> 
> technically, they are not queries but ndb.Future, so using f_ instead of q_
would make more sense.

Done.

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
File appengine/swarming/handlers_endpoints_test.py (right):

https://codereview.chromium.org/2198063002/diff/120001/appengine/swarming/han...
appengine/swarming/handlers_endpoints_test.py:1447: u'count': unicode(3),
On 2016/08/01 at 19:18:37, M-A Ruel wrote:
> Use actual values, it's shorter, e.g. u'3'

Done.

[kjlubick, 2016-08-01 19:25:45]

The CQ bit was checked by kjlubick@google.com

[kjlubick, 2016-08-01 19:25:46]

The patchset sent to the CQ was uploaded after l-g-t-m from maruel@chromium.org
Link to the patchset: https://codereview.chromium.org/2198063002/#ps140001
(title: "Address nits")

[commit-bot: I haz the power, 2016-08-01 19:25:52]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-01 19:29:08]

Description was changed from

==========
Add bots.count endpoint

This is analogous to tasks.count, with the addition of a count of dead,
quarantined and busy bots.

BUG=631047
==========

to

==========
Add bots.count endpoint

This is analogous to tasks.count, with the addition of a count of dead,
quarantined and busy bots.

BUG=631047

Committed:
https://github.com/luci/luci-py/commit/42b792562b6445d315416c6e306d24a71fa3626d
==========

[commit-bot: I haz the power, 2016-08-01 19:29:09]

Committed patchset #8 (id:140001) as
https://github.com/luci/luci-py/commit/42b792562b6445d315416c6e306d24a71fa3626d