client: Support repeated keys in task request.

client: Support repeated keys in task request.

- Add more flags to 'swarming.py bots': --idle, --busy, --mp, --non-mp
- Change 'swarming.py bots' dimensions filtering to be done server side.
- Refactor 'swarming.py bots' to reuse the same code as 'swarming.py query'.
- Support for the same key to be specified multiple times is needed for named
  caches, as unlike the other dimensions, it makes sense to specify the same key
  mulitple time.
- Only add support on client side, server side is to follow up shortly.

R=vadimsh@chromium.org
BUG=728124
Review-Url: https://codereview.chromium.org/2923633003
Committed: https://github.com/luci/luci-py/commit/1bd482d4014fd63fb2a263fd3e60b328f8be59ab

[M-A Ruel, 2017-06-05 20:30:54]

Description was changed from

==========
client: Support repeated keys in task request.

This is needed for named caches, as unlike the other dimensions, it makes sense
to specify the same key mulitple time.

Only add support on client side, server side is to follow up shortly.

Refactor 'swarming.py bots' to reuse the same code as 'swarming.py query'.

R=vadimsh@chromium.org
BUG=728124
==========

to

==========
client: Support repeated keys in task request.

- Add more flags to 'swarming.py bots': --idle, --busy, --mp, --non-mp
- Change 'swarming.py bots' dimensions filtering to be done server side.
- Refactor 'swarming.py bots' to reuse the same code as 'swarming.py query'.
- Support for the same key to be specified multiple times is needed for named
  caches, as unlike the other dimensions, it makes sense to specify the same key
  mulitple time.
- Only add support on client side, server side is to follow up shortly.

R=vadimsh@chromium.org
BUG=728124
==========

[Vadim Sh., 2017-06-05 21:33:13]

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py
File client/swarming.py (right):

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode849
client/swarming.py:849: return data, yielder
instead of returning data, you can yield data['items'] from yielder before doing
'while' loop there

In fact, I don't understand why this function returns a generator instead of
being a generator itself

def fetch_with_cursor(base_url, limit):
  ...
  yield data['items']
  ...
  while cursor and (....):
     ...
     yield items


At even closer look, it doesn't have to be generator at all. Just add 'progress'
argument and move the entire fetching loop here, instead of doing it in
individual subcommands.

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode1192
client/swarming.py:1192: url += '&dimensions=%s:%s' % (key, value)
this needs url encoding

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode1344
client/swarming.py:1344: # Some items support cursors. Try to get automatically
if cursors are
this comment doesn't seem relevant any more

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode1361
client/swarming.py:1361: tools.write_json(sys.stdout, data, False)
this progress output will screw up json

consider emitting it to stderr

[M-A Ruel, 2017-06-07 15:36:03]

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py
File client/swarming.py (right):

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode849
client/swarming.py:849: return data, yielder
On 2017/06/05 21:33:13, Vadim Sh. wrote:
> instead of returning data, you can yield data['items'] from yielder before
doing
> 'while' loop there
> 
> In fact, I don't understand why this function returns a generator instead of
> being a generator itself
> 
> def fetch_with_cursor(base_url, limit):
>   ...
>   yield data['items']
>   ...
>   while cursor and (....):
>      ...
>      yield items
> 
> 
> At even closer look, it doesn't have to be generator at all. Just add
'progress'
> argument and move the entire fetching loop here, instead of doing it in
> individual subcommands.

It's because some queries may not return 'items'. For example doing
swarming.py query task/1234/result

I added a more specific example in CMDquery and skipped the function when no
cursor or no items is provided.

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode1192
client/swarming.py:1192: url += '&dimensions=%s:%s' % (key, value)
On 2017/06/05 21:33:13, Vadim Sh. wrote:
> this needs url encoding

Argh, thanks. Changed to encode everything systematically.

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode1344
client/swarming.py:1344: # Some items support cursors. Try to get automatically
if cursors are
On 2017/06/05 21:33:13, Vadim Sh. wrote:
> this comment doesn't seem relevant any more

Done.

https://codereview.chromium.org/2923633003/diff/1/client/swarming.py#newcode1361
client/swarming.py:1361: tools.write_json(sys.stdout, data, False)
On 2017/06/05 21:33:13, Vadim Sh. wrote:
> this progress output will screw up json
> 
> consider emitting it to stderr

Done.

[Vadim Sh., 2017-06-07 17:14:26]

lgtm, but 'query' interface is weird. It's a mix of CLI interface and REST API
interface (users somehow should know what URLs to pass). Not super friendly (I
myself had to read the code each time I tried to use it to remember how exactly
to call it).

[M-A Ruel, 2017-06-07 18:10:39]

On 2017/06/07 17:14:26, Vadim Sh. wrote:
> lgtm, but 'query' interface is weird. It's a mix of CLI interface and REST API
> interface (users somehow should know what URLs to pass). Not super friendly (I
> myself had to read the code each time I tried to use it to remember how
exactly
> to call it).

I totally agree on the opaqueness of 'query'. :/

[M-A Ruel, 2017-06-08 13:24:32]

The CQ bit was checked by maruel@chromium.org

[commit-bot: I haz the power, 2017-06-08 13:24:42]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-06-08 13:26:53]

CQ is committing da patch.

Bot data: {"patchset_id": 20001, "attempt_start_ts": 1496928272844160,
"parent_rev": "f52c4527495dc0c4fbb42def00117214784afc44", "commit_rev":
"1bd482d4014fd63fb2a263fd3e60b328f8be59ab"}

[commit-bot: I haz the power, 2017-06-08 13:26:56]

Description was changed from

==========
client: Support repeated keys in task request.

- Add more flags to 'swarming.py bots': --idle, --busy, --mp, --non-mp
- Change 'swarming.py bots' dimensions filtering to be done server side.
- Refactor 'swarming.py bots' to reuse the same code as 'swarming.py query'.
- Support for the same key to be specified multiple times is needed for named
  caches, as unlike the other dimensions, it makes sense to specify the same key
  mulitple time.
- Only add support on client side, server side is to follow up shortly.

R=vadimsh@chromium.org
BUG=728124
==========

to

==========
client: Support repeated keys in task request.

- Add more flags to 'swarming.py bots': --idle, --busy, --mp, --non-mp
- Change 'swarming.py bots' dimensions filtering to be done server side.
- Refactor 'swarming.py bots' to reuse the same code as 'swarming.py query'.
- Support for the same key to be specified multiple times is needed for named
  caches, as unlike the other dimensions, it makes sense to specify the same key
  mulitple time.
- Only add support on client side, server side is to follow up shortly.

R=vadimsh@chromium.org
BUG=728124

Review-Url: https://codereview.chromium.org/2923633003
Committed:
https://github.com/luci/luci-py/commit/1bd482d4014fd63fb2a263fd3e60b328f8be59ab
==========

[commit-bot: I haz the power, 2017-06-08 13:26:57]

Committed patchset #2 (id:20001) as
https://github.com/luci/luci-py/commit/1bd482d4014fd63fb2a263fd3e60b328f8be59ab