Promote deterministic build CI to real CI (tree closer, except for the Clang builder)

Promote deterministic build CI to real CI (tree closer, except for the Clang builder)

BUG=496701

[Sébastien Marchand, 2015-06-04 20:26:45]

Patchset #1 (id:1) has been deleted

[Sébastien Marchand, 2015-06-04 20:26:57]

sebmarchand@chromium.org changed reviewers:
+ maruel@chromium.org

[Sébastien Marchand, 2015-06-04 20:26:57]

PTAL.

[M-A Ruel, 2015-06-04 20:56:08]

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:102: 'CrWinClang',
Move it there and name it
'CrWinClang Deterministic'

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:2005: b_win_clang_deterministic_build,
Move along other clang builders

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.fyi/slaves.cfg (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.fyi/slaves.cfg:931: {
Move along other crclang slaves

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.swarm/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.swarm/master.cfg:150: b_win_deterministic_build,
Remove these

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.win/master_win_cfg.py (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.win/master_win_cfg.py:93:
c['builders'].extend([b_win_deterministic_build])
c['builders'].append(b_win_deterministic_build)

[Sébastien Marchand, 2015-06-04 21:16:29]

Thanks, PTAnL.

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:102: 'CrWinClang',
On 2015/06/04 20:56:07, M-A Ruel wrote:
> Move it there and name it
> 'CrWinClang Deterministic'

Done.

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:2005: b_win_clang_deterministic_build,
On 2015/06/04 20:56:07, M-A Ruel wrote:
> Move along other clang builders

I've renamed this variable, otherwise it's already in the Clang Tot section.

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.fyi/slaves.cfg (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.fyi/slaves.cfg:931: {
On 2015/06/04 20:56:08, M-A Ruel wrote:
> Move along other crclang slaves

Done.

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.swarm/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.swarm/master.cfg:150: b_win_deterministic_build,
On 2015/06/04 20:56:08, M-A Ruel wrote:
> Remove these

Done.

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
File masters/master.chromium.win/master_win_cfg.py (right):

https://codereview.chromium.org/1147043007/diff/20001/masters/master.chromium...
masters/master.chromium.win/master_win_cfg.py:93:
c['builders'].extend([b_win_deterministic_build])
On 2015/06/04 20:56:08, M-A Ruel wrote:
> c['builders'].append(b_win_deterministic_build)

Done.

[M-A Ruel, 2015-06-04 21:30:33]

I cc'ed few people to alert them of the migration of the win clang deterministic
builder aside theirs on chromium.fyi. So they can follow up with this as they
desire. The VM is already preconfigured with cmake and is working already.

My hope is to get win clang to generate completely deterministic builds once Rui
is done with the linker job.

---

lgtm, it's odd of leaving the recipe in swarming/ but it still make sense. We
can migrate it but it requires copying, then removing the old ones.

Please only commit tomorrow morning so I can handle the fallout.

[Nico, 2015-06-04 21:37:26]

thakis@chromium.org changed reviewers:
+ thakis@chromium.org

[Nico, 2015-06-04 21:37:27]

having anything win/clang based be a tree closer does absolutely not lgtm at
this point. we don't have try bots, we started having prebuilts only last week,
etc.

[Nico, 2015-06-04 21:37:53]

ps: you shouldn't have cmake on that build, you should use the prebuild
binaries. ideally, that's happening automatically

[Nico, 2015-06-04 21:37:53]

ps: you shouldn't have cmake on that build, you should use the prebuild
binaries. ideally, that's happening automatically

[M-A Ruel, 2015-06-04 21:39:54]

On 2015/06/04 21:37:27, Nico wrote:
> having anything win/clang based be a tree closer does absolutely not lgtm at
> this point. we don't have try bots, we started having prebuilts only last
week,
> etc.

Please reread what I wrote, what Seb wrote. win/clang is on chromium.fyi. We're
simply moving it simultaneously with the others, the CL could have been clearer
on that.

[Nico, 2015-06-04 21:45:34]

Ah, ok, cool.

(Re comments: Check the output of
http://build.chromium.org/p/chromium.swarm/builders/Windows%20Clang%20determi...
to see that you're getting prebuilds. You don't need cmake on this bot.)

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1199: # Work around an issue with NaCl
build.
See all the "# not (clobber) cause crbug.com/399990" comments above; use a
shorter bot name instead

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1208: 'category': 'clang tot',
You should use the bundled clang and go in the "win clang" category instead

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.linux/master_linux_cfg.py (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.linux/master_linux_cfg.py:65: # Work around an issue
with NaCl build.
linux shouldn't need this

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.mac/master_mac_cfg.py (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.mac/master_mac_cfg.py:72: 'slavebuilddir': 'chr',
mac shouldn't need this either

[M-A Ruel, 2015-06-05 02:01:00]

Seb noted something I agree with, let's create the try builders first because
adding the CI ones. They would be set by default because they would be too slow
but it'd be good when someone wants to diagnose an issue. So let's postpone
moving the CI ones until the TS ones are created. The clang one could be
migrated immediately but it's easier to migrate all the chromium.swarm at once,
as this CL does. So the point is to create another CL about the TS that would be
committed first. Thanks.

[Sébastien Marchand, 2015-06-05 18:44:11]

Patchset #3 (id:60001) has been deleted

[Sébastien Marchand, 2015-06-05 18:44:42]

Rebased and addressed Nico's comments, PTAnL.

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1199: # Work around an issue with NaCl
build.
On 2015/06/04 21:45:33, Nico wrote:
> See all the "# not (clobber) cause crbug.com/399990" comments above; use a
> shorter bot name instead

imho it's cleaner to change the slavebuilddir argument instead of trying to find
a bot name that's "short enough".

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1208: 'category': 'clang tot',
On 2015/06/04 21:45:33, Nico wrote:
> You should use the bundled clang and go in the "win clang" category instead

Switched to "win clang", how can I make sure that I'm using the bundled clang ?
Is there a chromium/gclient recipe config flag that I should use or should it
just work ?

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.linux/master_linux_cfg.py (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.linux/master_linux_cfg.py:65: # Work around an issue
with NaCl build.
On 2015/06/04 21:45:33, Nico wrote:
> linux shouldn't need this

Done.

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.mac/master_mac_cfg.py (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.mac/master_mac_cfg.py:72: 'slavebuilddir': 'chr',
On 2015/06/04 21:45:33, Nico wrote:
> mac shouldn't need this either

Done.

[Nico, 2015-06-05 18:46:44]

lgtm

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1199: # Work around an issue with NaCl
build.
On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> On 2015/06/04 21:45:33, Nico wrote:
> > See all the "# not (clobber) cause crbug.com/399990" comments above; use a
> > shorter bot name instead
> 
> imho it's cleaner to change the slavebuilddir argument instead of trying to
find
> a bot name that's "short enough".

Maybe, but we should do the same everywhere. And I'm not sure if changing this
for the other bots will cause history loss.

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1208: 'category': 'clang tot',
On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> On 2015/06/04 21:45:33, Nico wrote:
> > You should use the bundled clang and go in the "win clang" category instead
> 
> Switched to "win clang", how can I make sure that I'm using the bundled clang
?
> Is there a chromium/gclient recipe config flag that I should use or should it
> just work ?

It should just work. The tot bots use the chromium_win_clang_tot config; as long
as you don't use that you're good.

[Sébastien Marchand, 2015-06-05 18:52:58]

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1199: # Work around an issue with NaCl
build.
On 2015/06/05 18:46:44, Nico wrote:
> On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> > On 2015/06/04 21:45:33, Nico wrote:
> > > See all the "# not (clobber) cause crbug.com/399990" comments above; use a
> > > shorter bot name instead
> > 
> > imho it's cleaner to change the slavebuilddir argument instead of trying to
> find
> > a bot name that's "short enough".
> 
> Maybe, but we should do the same everywhere. And I'm not sure if changing this
> for the other bots will cause history loss.

In this case, what's the definition of a "short" name ? (i.e. is "CrWinClang
Deterministic" short enough, or should it be CrWinClangDet ?)

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1208: 'category': 'clang tot',
On 2015/06/05 18:46:44, Nico wrote:
> On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> > On 2015/06/04 21:45:33, Nico wrote:
> > > You should use the bundled clang and go in the "win clang" category
instead
> > 
> > Switched to "win clang", how can I make sure that I'm using the bundled
clang
> ?
> > Is there a chromium/gclient recipe config flag that I should use or should
it
> > just work ?
> 
> It should just work. The tot bots use the chromium_win_clang_tot config; as
long
> as you don't use that you're good.

It's what we're using, we'll see if it works.

[Nico, 2015-06-05 18:58:43]

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1199: # Work around an issue with NaCl
build.
On 2015/06/05 18:52:57, Sébastien Marchand wrote:
> On 2015/06/05 18:46:44, Nico wrote:
> > On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> > > On 2015/06/04 21:45:33, Nico wrote:
> > > > See all the "# not (clobber) cause crbug.com/399990" comments above; use
a
> > > > shorter bot name instead
> > > 
> > > imho it's cleaner to change the slavebuilddir argument instead of trying
to
> > find
> > > a bot name that's "short enough".
> > 
> > Maybe, but we should do the same everywhere. And I'm not sure if changing
this
> > for the other bots will cause history loss.
> 
> In this case, what's the definition of a "short" name ? (i.e. is "CrWinClang
> Deterministic" short enough, or should it be CrWinClangDet ?)

You have 18 characters. "ClangDeterministic" is 18, 'ClangWinRepeatable' is 18,
etc

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1208: 'category': 'clang tot',
On 2015/06/05 18:52:57, Sébastien Marchand wrote:
> On 2015/06/05 18:46:44, Nico wrote:
> > On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> > > On 2015/06/04 21:45:33, Nico wrote:
> > > > You should use the bundled clang and go in the "win clang" category
> instead
> > > 
> > > Switched to "win clang", how can I make sure that I'm using the bundled
> clang
> > ?
> > > Is there a chromium/gclient recipe config flag that I should use or should
> it
> > > just work ?
> > 
> > It should just work. The tot bots use the chromium_win_clang_tot config; as
> long
> > as you don't use that you're good.
> 
> It's what we're using, we'll see if it works.

https://codereview.chromium.org/1147043007/diff/80001/scripts/slave/recipes/s...
says `'chromium_config': 'chromium_win_clang',` (note no _tot suffix), so I
don't think you use the tot config. (Maybe I misunderstood what "It" referred
to)

[Sébastien Marchand, 2015-06-05 19:09:07]

One last look ? Please DO NOT flip the CQ bit, I'm waiting for the try server
masters to be restarted (I want to make sure that the new builders work) before
landing this.

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
File masters/master.chromium.fyi/master.cfg (right):

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1199: # Work around an issue with NaCl
build.
On 2015/06/05 18:58:43, Nico wrote:
> On 2015/06/05 18:52:57, Sébastien Marchand wrote:
> > On 2015/06/05 18:46:44, Nico wrote:
> > > On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> > > > On 2015/06/04 21:45:33, Nico wrote:
> > > > > See all the "# not (clobber) cause crbug.com/399990" comments above;
use
> a
> > > > > shorter bot name instead
> > > > 
> > > > imho it's cleaner to change the slavebuilddir argument instead of trying
> to
> > > find
> > > > a bot name that's "short enough".
> > > 
> > > Maybe, but we should do the same everywhere. And I'm not sure if changing
> this
> > > for the other bots will cause history loss.
> > 
> > In this case, what's the definition of a "short" name ? (i.e. is "CrWinClang
> > Deterministic" short enough, or should it be CrWinClangDet ?)
> 
> You have 18 characters. "ClangDeterministic" is 18, 'ClangWinRepeatable' is
18,
> etc

ClangWinRepeatable it is.

https://codereview.chromium.org/1147043007/diff/40001/masters/master.chromium...
masters/master.chromium.fyi/master.cfg:1208: 'category': 'clang tot',
On 2015/06/05 18:58:43, Nico wrote:
> On 2015/06/05 18:52:57, Sébastien Marchand wrote:
> > On 2015/06/05 18:46:44, Nico wrote:
> > > On 2015/06/05 18:44:42, Sébastien Marchand wrote:
> > > > On 2015/06/04 21:45:33, Nico wrote:
> > > > > You should use the bundled clang and go in the "win clang" category
> > instead
> > > > 
> > > > Switched to "win clang", how can I make sure that I'm using the bundled
> > clang
> > > ?
> > > > Is there a chromium/gclient recipe config flag that I should use or
should
> > it
> > > > just work ?
> > > 
> > > It should just work. The tot bots use the chromium_win_clang_tot config;
as
> > long
> > > as you don't use that you're good.
> > 
> > It's what we're using, we'll see if it works.
> 
>
https://codereview.chromium.org/1147043007/diff/80001/scripts/slave/recipes/s...
> says `'chromium_config': 'chromium_win_clang',` (note no _tot suffix), so I
> don't think you use the tot config. (Maybe I misunderstood what "It" referred
> to)

Oh sorry, I missed the '_tot' suffix. Fixing this.

[Nico, 2015-06-05 19:12:45]

https://codereview.chromium.org/1147043007/diff/140001/scripts/slave/recipes/...
File scripts/slave/recipes/deterministic_build.py (right):

https://codereview.chromium.org/1147043007/diff/140001/scripts/slave/recipes/...
scripts/slave/recipes/deterministic_build.py:49: 'chromium_config':
'chromium_win_clang_tot',
I was trying to say "don't use _tot". The _tot bots build clang tot, require
cmake, and are a lot less reliable (since upstream clang is red every now and
then). Use category "win clang" and config 'chromium_win_clang'.

[Sébastien Marchand, 2015-06-05 19:44:45]

https://codereview.chromium.org/1147043007/diff/140001/scripts/slave/recipes/...
File scripts/slave/recipes/deterministic_build.py (right):

https://codereview.chromium.org/1147043007/diff/140001/scripts/slave/recipes/...
scripts/slave/recipes/deterministic_build.py:49: 'chromium_config':
'chromium_win_clang_tot',
On 2015/06/05 19:12:45, Nico wrote:
> I was trying to say "don't use _tot". The _tot bots build clang tot, require
> cmake, and are a lot less reliable (since upstream clang is red every now and
> then). Use category "win clang" and config 'chromium_win_clang'.

Sorry, I've misread your comment. I've switched back to chromium_win_clang.

[Nico, 2015-06-05 19:46:50]

fyi lgtm