swarming: read configs from luci-config

swarming: read configs from luci-config

- add proto/config.proto with SettingsCfg message that has same field as
  GlobalConfig.
- add a cron job that updates settings.cfg in the background.
- add ConfigApi so we can specify hostname of config service in a
  particular swarming instance.
- update config.py to read settings using components.config instead of
  GlobalConfig entity. Remove GlobalConfig.
- update handlers.py and restricted_config.html accordingly
  Remove code related to config mutation.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=601018
Committed: https://github.com/luci/luci-py/commit/fd161432f88fcfa251372ca3e1a3e45c49d24216

[nodir, 2016-04-27 17:22:11]

Description was changed from

==========
swarming: read configs from luci-config

- add proto/config.proto with SettingsCfg message that has same field as
  GlobalConfig.
- add a cron job that updates settings.cfg in the background.
- add ConfigApi so we can specify hostname of config service in a
  particular swarming instance.
- update config.py to read settings using components.config instead of
  GlobalConfig entity
- update handlers.py and restricted_config.html accordingly
  Remove code related to config mutation.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=601018
==========

to

==========
swarming: read configs from luci-config

- add proto/config.proto with SettingsCfg message that has same field as
  GlobalConfig.
- add a cron job that updates settings.cfg in the background.
- add ConfigApi so we can specify hostname of config service in a
  particular swarming instance.
- update config.py to read settings using components.config instead of
  GlobalConfig entity. Remove GlobalConfig.
- update handlers.py and restricted_config.html accordingly
  Remove code related to config mutation.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=601018
==========

[nodir, 2016-04-27 17:22:28]

The CQ bit was checked by nodir@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-04-27 17:22:41]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1926443003/20001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1926443003/20001

[nodir, 2016-04-27 17:22:48]

PTAL

Config: https://chromereviews.googleplex.com/410487013/

[commit-bot: I haz the power, 2016-04-27 17:28:49]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-27 17:28:50]

Dry run: Try jobs failed on following builders:
  Luci-py Presubmit on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-py%20Presubmit/bui...)

[nodir, 2016-04-27 17:35:04]

The CQ bit was checked by nodir@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-04-27 17:35:12]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1926443003/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1926443003/40001

[commit-bot: I haz the power, 2016-04-27 17:38:03]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-27 17:38:04]

Dry run: This issue passed the CQ dry run.

[M-A Ruel, 2016-04-27 17:50:46]

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/main.py
File appengine/swarming/main.py (right):

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/main...
appengine/swarming/main.py:42: # App that serves new endpoints API.
Remove this comment, it was stale.

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
File appengine/swarming/server/config.py (right):

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
appengine/swarming/server/config.py:29:
@utils.memcache('config:get_configs_url', time=300)
5 minutes is a long time. 1 minute would be more sensible.

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
appengine/swarming/server/config.py:70: )
This doesn't match the code style of this project.

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/serv...
File appengine/swarming/server/config.py (right):

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/serv...
appengine/swarming/server/config.py:26: """Validates settings.cfg file against
proto message schema."""
Assert bot_death_timeout_secs and reusable_task_age_secs make sense? >= 0 and <=
1 year. I realize this wasn't verified before so no biggies.

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/serv...
appengine/swarming/server/config.py:68: reusable_task_age_secs=7*24*60*60,
Could these be set as default in the proto instead?

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/temp...
File appengine/swarming/templates/restricted_config.html (right):

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/temp...
appengine/swarming/templates/restricted_config.html:16: <a
href="https://luci-config.appspot.com">luci-config</a>.
The path can't be hardcoded.

[nodir, 2016-04-27 18:37:22]

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/main.py
File appengine/swarming/main.py (right):

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/main...
appengine/swarming/main.py:42: # App that serves new endpoints API.
On 2016/04/27 17:50:45, M-A Ruel wrote:
> Remove this comment, it was stale.

Done.

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
File appengine/swarming/server/config.py (right):

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
appengine/swarming/server/config.py:29:
@utils.memcache('config:get_configs_url', time=300)
On 2016/04/27 17:50:45, M-A Ruel wrote:
> 5 minutes is a long time. 1 minute would be more sensible.

This value almost never changes.

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
appengine/swarming/server/config.py:70: )
On 2016/04/27 17:50:45, M-A Ruel wrote:
> This doesn't match the code style of this project.

you mean that the closing bracket must be on the same line?

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/serv...
File appengine/swarming/server/config.py (right):

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/serv...
appengine/swarming/server/config.py:26: """Validates settings.cfg file against
proto message schema."""
On 2016/04/27 17:50:46, M-A Ruel wrote:
> Assert bot_death_timeout_secs and reusable_task_age_secs make sense? >= 0 and
<=
> 1 year. I realize this wasn't verified before so no biggies.

Done.

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/serv...
appengine/swarming/server/config.py:68: reusable_task_age_secs=7*24*60*60,
On 2016/04/27 17:50:46, M-A Ruel wrote:
> Could these be set as default in the proto instead?

proto3 got rid of default values because it was a bad idea (because proto is
only a protocol spec and not more), so let's not repeat other's mistakes

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/temp...
File appengine/swarming/templates/restricted_config.html (right):

https://codereview.chromium.org/1926443003/diff/60001/appengine/swarming/temp...
appengine/swarming/templates/restricted_config.html:16: <a
href="https://luci-config.appspot.com">luci-config</a>.
On 2016/04/27 17:50:46, M-A Ruel wrote:
> The path can't be hardcoded.

Done.

[nodir, 2016-04-27 18:37:28]

The CQ bit was checked by nodir@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-04-27 18:37:34]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1926443003/120001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1926443003/120001

[commit-bot: I haz the power, 2016-04-27 18:40:04]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-27 18:40:05]

Dry run: This issue passed the CQ dry run.

[M-A Ruel, 2016-04-27 18:51:19]

lgtm

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
File appengine/swarming/server/config.py (right):

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
appengine/swarming/server/config.py:29:
@utils.memcache('config:get_configs_url', time=300)
On 2016/04/27 18:37:21, nodir wrote:
> On 2016/04/27 17:50:45, M-A Ruel wrote:
> > 5 minutes is a long time. 1 minute would be more sensible.
> 
> This value almost never changes.

but when it does, we don't want to wait. Furthermore, getting "settings
toggling" as one gets older, newer, then older again can be potentially
annoying/harmful. 1 minute is more reasonable.

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
appengine/swarming/server/config.py:70: )
On 2016/04/27 18:37:21, nodir wrote:
> On 2016/04/27 17:50:45, M-A Ruel wrote:
> > This doesn't match the code style of this project.
> 
> you mean that the closing bracket must be on the same line?

yes

[nodir, 2016-04-27 19:08:53]

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
File appengine/swarming/server/config.py (right):

https://codereview.chromium.org/1926443003/diff/40001/appengine/swarming/serv...
appengine/swarming/server/config.py:29:
@utils.memcache('config:get_configs_url', time=300)
On 2016/04/27 18:51:19, M-A Ruel wrote:
> On 2016/04/27 18:37:21, nodir wrote:
> > On 2016/04/27 17:50:45, M-A Ruel wrote:
> > > 5 minutes is a long time. 1 minute would be more sensible.
> > 
> > This value almost never changes.
> 
> but when it does, we don't want to wait. Furthermore, getting "settings
> toggling" as one gets older, newer, then older again can be potentially
> annoying/harmful. 1 minute is more reasonable.

changed to 1min

[nodir, 2016-04-27 20:47:24]

I made settings.cfg optional because other instances of swarming use default
values

[nodir, 2016-04-27 21:22:13]

The CQ bit was checked by nodir@chromium.org

[nodir, 2016-04-27 21:22:13]

The patchset sent to the CQ was uploaded after l-g-t-m from maruel@chromium.org
Link to the patchset: https://codereview.chromium.org/1926443003/#ps160001
(title: "changed logging levels")

[commit-bot: I haz the power, 2016-04-27 21:22:22]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1926443003/160001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1926443003/160001

[commit-bot: I haz the power, 2016-04-27 21:25:21]

Description was changed from

==========
swarming: read configs from luci-config

- add proto/config.proto with SettingsCfg message that has same field as
  GlobalConfig.
- add a cron job that updates settings.cfg in the background.
- add ConfigApi so we can specify hostname of config service in a
  particular swarming instance.
- update config.py to read settings using components.config instead of
  GlobalConfig entity. Remove GlobalConfig.
- update handlers.py and restricted_config.html accordingly
  Remove code related to config mutation.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=601018
==========

to

==========
swarming: read configs from luci-config

- add proto/config.proto with SettingsCfg message that has same field as
  GlobalConfig.
- add a cron job that updates settings.cfg in the background.
- add ConfigApi so we can specify hostname of config service in a
  particular swarming instance.
- update config.py to read settings using components.config instead of
  GlobalConfig entity. Remove GlobalConfig.
- update handlers.py and restricted_config.html accordingly
  Remove code related to config mutation.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=601018

Committed:
https://github.com/luci/luci-py/commit/fd161432f88fcfa251372ca3e1a3e45c49d24216
==========

[commit-bot: I haz the power, 2016-04-27 21:25:22]

Committed patchset #9 (id:160001) as
https://github.com/luci/luci-py/commit/fd161432f88fcfa251372ca3e1a3e45c49d24216