Adjust master_gen.PopulateBuildmasterConfig() to look at master_site_config

Adjust master_gen.PopulateBuildmasterConfig() to look at master_site_config.
    
When master_gen was initially written, we had hoped to be able to delete
master_site_config, and so master.cfg would call
master_gen.PopulateBuildmasterConfig() to define the active_master class
on the fly.
    
However, we realized that we still needed master_site_config.py to exist
for various other scripts and tests, even though the buildbot startup
itself didn't need it. So, we added a template to create the site_config
as well. Unfortunately, the code was now duplicated in two places and
potentially not in sync. In particular, the bug below was caused by
master_site_config being updated, but master_gen not being updated.
    
This CL changes the master.cfg template to actually use the
master_site_config class definition, and simplifies master_gen as
a result, so that the class definitions aren't duplicated.
    
Generally, we don't want master.cfg to import any other modules, because
every module that is imported will *not* be reload on 'make config',
and if that module changes we will actually need to restart the
master. However, the fields that are defined in master_site_config.py,
while pulled from builders.pyl, are generally write-once, defined when
the master is initially created, and shouldn't change over the lifetime
of the master. I guess we'll see if that ends up being true in practice.
    
R=nodir@chromium.org, luqui@chromium.org
BUG=490830
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=295381

[Dirk Pranke, 2015-05-21 23:29:06]

dpranke@chromium.org changed reviewers:
+ luqui@chromium.org, nodir@chromium.org

[Dirk Pranke, 2015-05-21 23:31:29]

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.py
File scripts/master/master_gen.py (left):

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:44: # without needing the master to re-read things.
luqui@ fixed this TODO in https://codereview.chromium.org/860383008/

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.py
File scripts/master/master_gen.py (right):

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:34: c['projectURL'] = Master.project_url
master_cls is only ever not equal to Master in the unit tests, and it turns out
that using Master in the unit tests for this works just fine as well.

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:75: Master.GetBotPassword(),
same comment as line 34.

[nodir, 2015-05-21 23:44:35]

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.py
File scripts/master/master_gen.py (right):

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:29: m_annotator =
annotator_factory.AnnotatorFactory()
pass active_master_cls as a parameter,
https://code.google.com/p/chromium/codesearch#chromium/build/scripts/master/f...
It is required for cross-master triggering

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:34: c['projectURL'] = Master.project_url
On 2015/05/21 23:31:29, Dirk Pranke wrote:
> master_cls is only ever not equal to Master in the unit tests, and it turns
out
> that using Master in the unit tests for this works just fine as well.

So all masters have the same project_url?
As a user of builders.pyl, I'd expect to be able to specify in the builders.pyl.

[Dirk Pranke, 2015-05-21 23:57:31]

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.py
File scripts/master/master_gen.py (right):

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:29: m_annotator =
annotator_factory.AnnotatorFactory()
On 2015/05/21 23:44:35, nodir wrote:
> pass active_master_cls as a parameter,
>
https://code.google.com/p/chromium/codesearch#chromium/build/scripts/master/f...
> It is required for cross-master triggering

Does cross-master triggering require anything else from the buildbot config (or
builders.pyl), or is the rest of it completely handled inside the recipe?

I can add the class here, of course, but I want to make sure we know and can
keep track of what configs are supported.

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:34: c['projectURL'] = Master.project_url
On 2015/05/21 23:44:34, nodir wrote:
> On 2015/05/21 23:31:29, Dirk Pranke wrote:
> > master_cls is only ever not equal to Master in the unit tests, and it turns
> out
> > that using Master in the unit tests for this works just fine as well.
> 
> So all masters have the same project_url?
> As a user of builders.pyl, I'd expect to be able to specify in the
builders.pyl.

That is a reasonable request, but the code doesn't currently support it (i.e.,
this change isn't changing that). I would just as soon not add it in this CL, to
avoid confusion. Feel free to file a separate bug/feature request.

[nodir, 2015-05-22 00:09:28]

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.py
File scripts/master/master_gen.py (right):

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:29: m_annotator =
annotator_factory.AnnotatorFactory()
On 2015/05/21 23:57:31, Dirk Pranke wrote:
> On 2015/05/21 23:44:35, nodir wrote:
> > pass active_master_cls as a parameter,
> >
>
https://code.google.com/p/chromium/codesearch#chromium/build/scripts/master/f...
> > It is required for cross-master triggering
> 
> Does cross-master triggering require anything else from the buildbot config
(or
> builders.pyl), or is the rest of it completely handled inside the recipe?

This should be enough. The rest is handled inside annotator engine and recipes
 
> I can add the class here, of course, but I want to make sure we know and can
> keep track of what configs are supported.

Passing it won't do any harm: it just assigns an attribute.
It is used only if someone attempts to trigger a cross-master build, or a
buildbucket-based build triggers a new build

https://codereview.chromium.org/1152093002/diff/1/scripts/master/master_gen.p...
scripts/master/master_gen.py:34: c['projectURL'] = Master.project_url
On 2015/05/21 23:57:31, Dirk Pranke wrote:
> On 2015/05/21 23:44:34, nodir wrote:
> > On 2015/05/21 23:31:29, Dirk Pranke wrote:
> > > master_cls is only ever not equal to Master in the unit tests, and it
turns
> > out
> > > that using Master in the unit tests for this works just fine as well.
> > 
> > So all masters have the same project_url?
> > As a user of builders.pyl, I'd expect to be able to specify in the
> builders.pyl.
> 
> That is a reasonable request, but the code doesn't currently support it (i.e.,
> this change isn't changing that). I would just as soon not add it in this CL,
to
> avoid confusion. Feel free to file a separate bug/feature request.

Acknowledged.

[Dirk Pranke, 2015-05-22 00:41:56]

parameter added. please take another look?

[nodir, 2015-05-22 00:54:54]

lgtm

[Dirk Pranke, 2015-05-22 00:59:47]

Thanks, Nodir!

Luqui@, feel like looking as well?

[luqui, 2015-05-22 04:38:58]

lgtm also.

[Dirk Pranke, 2015-05-22 16:30:20]

The CQ bit was checked by dpranke@chromium.org

[commit-bot: I haz the power, 2015-05-22 16:30:32]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1152093002/20001

[commit-bot: I haz the power, 2015-05-22 16:34:43]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-22 16:34:44]

Presubmit check for 1152093002-20001 failed and returned exit status 1.

Running presubmit commit checks ...

** Presubmit ERRORS **
tests/masters_test.py (91.85s) failed
1 masters failed:
master.client.catapult
69 masters succeeded, 1 failed, 0 skipped in 91.7s.

=== Error running master.client.catapult === 
2015-05-22 09:31:09-0700 [-] buildbot.status.web.baseweb.RotateLogSite starting
on 20303
2015-05-22 09:31:09-0700 [-] Starting factory
<buildbot.status.web.baseweb.RotateLogSite instance at 0x2ac6bce4b0e0>
2015-05-22 09:31:09-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2015-05-22 09:31:09-0700 [-] WebStatus using
(/b/infra_internal/commit_queue/workdir/tools/build/masters/master.chromium/public_html)
2015-05-22 09:31:09-0700 [-] adding IStatusReceiver <WebStatus on port tcp:40303
at 0x2ac6bc2e5ef0>
2015-05-22 09:31:09-0700 [-] Unhandled Error
	Traceback (most recent call last):
	  File
"/b/infra_internal/commit_queue/workdir/tools/build/third_party/twisted_10_2/twisted/application/service.py",
line 184, in setServiceParent
	    self.parent.addService(self)
	  File
"/b/infra_internal/commit_queue/workdir/tools/build/third_party/twisted_10_2/twisted/application/service.py",
line 303, in addService
	    service.privilegedStartService()
	  File
"/b/infra_internal/commit_queue/workdir/tools/build/third_party/twisted_10_2/twisted/application/internet.py",
line 357, in privilegedStartService
	    self._waitingForPort = self.endpoint.listen(self.factory)
	  File
"/b/infra_internal/commit_queue/workdir/tools/build/third_party/twisted_10_2/twisted/internet/endpoints.py",
line 187, in listen
	    interface=self._interface)
	--- <exception caught here> ---
	  File
"/b/infra_internal/commit_queue/workdir/tools/build/third_party/twisted_10_2/twisted/internet/defer.py",
line 104, in execute
	    result = callable(*args, **kw)
	  File
"/b/infra_internal/commit_queue/workdir/tools/build/third_party/twisted_10_2/twisted/internet/posixbase.py",
line 419, in listenTCP
	    p.startListening()
	  File
"/b/infra_internal/commit_queue/workdir/tools/build/third_party/twisted_10_2/twisted/internet/tcp.py",
line 857, in startListening
	    raise CannotListenError, (self.interface, self.port, le)
	twisted.internet.error.CannotListenError: Couldn't listen on any:40303: [Errno
98] Address already in use.



Presubmit checks took 243.7s to calculate.

[sullivan, 2015-05-22 17:01:42]

The CQ bit was checked by sullivan@chromium.org

[commit-bot: I haz the power, 2015-05-22 17:01:57]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1152093002/20001

[commit-bot: I haz the power, 2015-05-22 17:06:17]

Committed patchset #2 (id:20001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=295381