CrOS: Update public waterfall to auto-configure.

CrOS: Update public waterfall to auto-configure.

Update the 'chromiumos' public waterfall to auto-configure from the
pinned Chromite instance.

This change modifies the builder directory names on the master to
conform to a global CrOS standard. When this commit is implemented,
build directories will have to be updated on master to preserve builder
data.

BUG=chromium:456203
TEST=local
  - Started master, confirmed that builder population and slave
    allocation is correct.

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=294775

[dnj, 2015-04-08 23:01:08]

dnj@chromium.org changed reviewers:
+ petermayo@chromium.org, smut@google.com, stip@chromium.org

[dnj, 2015-04-08 23:01:08]

PTAL!

This change converts the "chromiumos" master over to use Chromite-driven
configuration.

Don't focus to much on the cros_builder_convert.py script. It's a transition
utility script to document/check build directory conversions. I'm going to use
it for this waterfall and probably one more and then delete it.

[ghost stip (do not use), 2015-04-08 23:21:18]

this is kind of a large, complicated change. can you break it up at all?
specifically, I see that you're making changes to the master.cfg (load things up
using a new config) and slaves.cfg (adjust how slave pools are defined) as well
as many other things. if you could split those up into separate changes it would
be far easier to review. I understand that different master configs make for
different slave configs, but I think this is a good path to go down for
untangling this CL

[dnj, 2015-04-08 23:22:40]

On 2015/04/08 23:21:18, stip wrote:
> this is kind of a large, complicated change. can you break it up at all?
> specifically, I see that you're making changes to the master.cfg (load things
up
> using a new config) and slaves.cfg (adjust how slave pools are defined) as
well
> as many other things. if you could split those up into separate changes it
would
> be far easier to review. I understand that different master configs make for
> different slave configs, but I think this is a good path to go down for
> untangling this CL

I can't really break that up. The master config and the slave allocation are all
intermingled.

This isn't unprecedented, though. The internal "chromeos.release" master has
already undergone a similar transformation, so a lot of the semantics and
structures are mirrored there.

[dnj, 2015-04-10 18:43:54]

pang

[chromium-reviews, 2015-04-10 19:14:49]

But how can I review if I can't comment <https://crbug.com/476045>?

On Fri, Apr 10, 2015 at 11:43 AM, <dnj@chromium.org> wrote:

> pang
>
> https://codereview.chromium.org/1068263003/
>

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[smut, 2015-04-10 21:17:17]

Fuck it. Lgtm.

But seriously, this generally looks fine (% tiny comments), although it seems
like cros_builder_convert.py could've been in a separate CL.

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/c...
File masters/master.chromiumos/chromiumos_board_config.py (right):

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/c...
masters/master.chromiumos/chromiumos_board_config.py:8: from collections import
OrderedDict, namedtuple
import collections (and group with logging, sys)

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/m...
File masters/master.chromiumos/master.cfg (right):

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/m...
masters/master.chromiumos/master.cfg:105: kwargs['max_time'] = 22*3600
Consider a constant at the top. And does this really take 22 hours?

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/s...
File masters/master.chromiumos/slaves.cfg (right):

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/s...
masters/master.chromiumos/slaves.cfg:5: from chromiumos_board_config import *
Why not just import chromiumos_board_config, and then refer to members with the
module. prefix? Makes it easier to trace where things are defined. Especially in
favor of importing *, because then there's no record of where certain objects
are coming from anywhere in this file.

[dnj, 2015-04-10 21:42:45]

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/c...
File masters/master.chromiumos/chromiumos_board_config.py (right):

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/c...
masters/master.chromiumos/chromiumos_board_config.py:8: from collections import
OrderedDict, namedtuple
On 2015/04/10 21:17:17, smut wrote:
> import collections (and group with logging, sys)

Done.

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/m...
File masters/master.chromiumos/master.cfg (right):

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/m...
masters/master.chromiumos/master.cfg:105: kwargs['max_time'] = 22*3600
On 2015/04/10 21:17:17, smut wrote:
> Consider a constant at the top. And does this really take 22 hours?

Good idea - I'm going to pull this out as a builder-type property instead.

And it can definitely take 14+ hours, don't know about 22. It's a pretty silly
builder.

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/s...
File masters/master.chromiumos/slaves.cfg (right):

https://codereview.chromium.org/1068263003/diff/1/masters/master.chromiumos/s...
masters/master.chromiumos/slaves.cfg:5: from chromiumos_board_config import *
On 2015/04/10 21:17:17, smut wrote:
> Why not just import chromiumos_board_config, and then refer to members with
the
> module. prefix? Makes it easier to trace where things are defined. Especially
in
> favor of importing *, because then there's no record of where certain objects
> are coming from anywhere in this file.

Mostly because "chromiumos_board_config.builder_configs" is long. I'll go ahead
and do this.

[ghost stip (do not use), 2015-04-10 22:47:04]

untangling large CLs makes it easier for a reviewer to spot errors or
inefficiencies. I'm confident this could have been made as a series of smaller
incremental CLs but this has already been written so lgtm

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
File masters/master.chromiumos/chromiumos_board_config.py (right):

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/chromiumos_board_config.py:1: # Copyright (c) 2015 The
Chromium Authors. All rights reserved.
reviewer note: this file is used by the master to enumerate all the configs from
chromite

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
File masters/master.chromiumos/master.cfg (left):

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:124: def _AddBoardTarget(board,
target_type, **kw):
review note: all these have been replaced by
chromiumos_board_config.builder_configs

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
File masters/master.chromiumos/master.cfg (right):

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:48: config.DatabaseSetup(c,
require_dbconfig=ActiveMaster.is_production_host)
review note: this is from line 183

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:107: factory =
chromeos_factory.CbuildbotFactory(
review note: line 67

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:108: params=str(bc.config.name),
branch=branch, buildroot=cbuild_root,
review note: 'config' in getcbuildbotfactory (line 65) is now bc.config.name,
which must be converted to string for some reason

branch is a new parameter

buildroot used to be hardcoded to /b/cbuild/external_master, now is cbuild_root
which is slave_root/cbuild/root_dir    ->  root_dir is passed into GenCBuild

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:109: show_gclient_output=False,
dry_run=DRY_RUN, **kwargs
review note: DRY_RUN is line 42, same as 'not activemaster' on line 68

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:112: builder = {
review note: GetCBuildbotFactory was called by AddBuilderDefinition, these are
both now collapsed into GenCBuild

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:266: # Save our slave pool state. This is
populated when our 'slaves' variable
reviewer note: this is all for when is_production_host is false

https://chromiumcodereview.appspot.com/1068263003/diff/20001/masters/master.c...
masters/master.chromiumos/master.cfg:277: # Disable 'auto_reboot' on slaves for
local testing.
weird, I was under the impression  that the slave itself would detect
non-production and not reboot. guess not

https://chromiumcodereview.appspot.com/1068263003/diff/20001/scripts/common/c...
File scripts/common/cros_chromite.py (right):

https://chromiumcodereview.appspot.com/1068263003/diff/20001/scripts/common/c...
scripts/common/cros_chromite.py:75: ASAN = 'asan'
reviewer note: this is not merely an alphabetic change. ASAN is a new value

https://chromiumcodereview.appspot.com/1068263003/diff/20001/scripts/common/s...
File scripts/common/slave_alloc.py (right):

https://chromiumcodereview.appspot.com/1068263003/diff/20001/scripts/common/s...
scripts/common/slave_alloc.py:123: state = {}
why is this not None?

https://chromiumcodereview.appspot.com/1068263003/diff/20001/scripts/tools/cr...
File scripts/tools/cros/cros_builder_convert.py (right):

https://chromiumcodereview.appspot.com/1068263003/diff/20001/scripts/tools/cr...
scripts/tools/cros/cros_builder_convert.py:29: """
I'm told by dan that this is a one-shot script

[dnj, 2015-04-12 15:20:38]

https://codereview.chromium.org/1068263003/diff/20001/masters/master.chromium...
File masters/master.chromiumos/master.cfg (right):

https://codereview.chromium.org/1068263003/diff/20001/masters/master.chromium...
masters/master.chromiumos/master.cfg:277: # Disable 'auto_reboot' on slaves for
local testing.
On 2015/04/10 22:47:03, stip wrote:
> weird, I was under the impression  that the slave itself would detect
> non-production and not reboot. guess not

I'm pretty sure it does, as I've never had a system reboot when running a slave.
Maybe they didn't originally? All of these waterfalls could use a cleanup.

https://codereview.chromium.org/1068263003/diff/20001/scripts/common/slave_al...
File scripts/common/slave_alloc.py (right):

https://codereview.chromium.org/1068263003/diff/20001/scripts/common/slave_al...
scripts/common/slave_alloc.py:123: state = {}
On 2015/04/10 22:47:04, stip wrote:
> why is this not None?

This handles the case where a state file does not exist.

https://codereview.chromium.org/1068263003/diff/20001/scripts/tools/cros/cros...
File scripts/tools/cros/cros_builder_convert.py (right):

https://codereview.chromium.org/1068263003/diff/20001/scripts/tools/cros/cros...
scripts/tools/cros/cros_builder_convert.py:29: """
On 2015/04/10 22:47:04, stip wrote:
> I'm told by dan that this is a one-shot script

One shot per master, but versioned / exists b/c I want something better than
hand tailoring the directories on live master.

[dnj, 2015-04-13 04:10:05]

The CQ bit was checked by dnj@chromium.org

[dnj, 2015-04-13 04:10:05]

The patchset sent to the CQ was uploaded after l-g-t-m from smut@google.com,
stip@chromium.org
Link to the patchset: https://codereview.chromium.org/1068263003/#ps40001
(title: "Added experimental/documentation annotations to web UI.")

[commit-bot: I haz the power, 2015-04-13 04:10:12]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1068263003/40001

[commit-bot: I haz the power, 2015-04-13 04:14:41]

Committed patchset #3 (id:40001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=294775