Add linux_clang to default try jobs

Add linux_clang to default try jobs

R=thakis@chromium.org
BUG=
TEST=
NOTRY=true

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=125922

[Nico, 2011-07-26 19:52:11]

How many default try jobs are sent per day? How many builds does the linux_clang
bot do currently per day?

[M-A Ruel, 2011-07-26 21:27:52]

On 2011/07/26 19:52:11, Nico wrote:
> How many default try jobs are sent per day?

./buildbot_json.py interactive build.chromium.org/p/tryserver.chromium
>>> today = [i for i in b.builders['win'].builds if i.data['times'][0] >=
time.time() - 24*60*60]

>>> len(today)
321

>>> len([i for i in today if 'commit-bot@chromium.org' not in i.blame])
192

So 321 - 192 = 129 try jobs by the commit bot.

> How many builds does the linux_clang
> bot do currently per day?

24h/d * 60m/h * 7 builders / 30/m/build = 336 builds/day

[Nico, 2011-07-26 21:34:04]

On 2011/07/26 21:27:52, Marc-Antoine Ruel wrote:
> On 2011/07/26 19:52:11, Nico wrote:
> > How many default try jobs are sent per day?
> 
> ./buildbot_json.py interactive build.chromium.org/p/tryserver.chromium
> >>> today = [i for i in b.builders['win'].builds if i.data['times'][0] >=
> time.time() - 24*60*60]
> 
> >>> len(today)
> 321
> 
> >>> len([i for i in today if 'commit-bot@chromium.org' not in i.blame])
> 192
> 
> So 321 - 192 = 129 try jobs by the commit bot.
> 
> > How many builds does the linux_clang
> > bot do currently per day?
> 
> 24h/d * 60m/h * 7 builders / 30/m/build = 336 builds/day

That assumes a  uniform distribution though :-)

Looks like
http://build.chromium.org/p/tryserver.chromium/builders/linux_clang/builds/5058
was roughly 1 day ago, so the bot does roughly 130 builds per day currently and
is barely able to keep up during peak hours.

This change increases load by 148%, which suggests we will need several more
slaves (5-7?), lest we slow down the commit-bot. Do we have the resources for
this?

[M-A Ruel, 2011-07-27 15:14:37]

On 2011/07/26 21:34:04, Nico wrote:
> This change increases load by 148%, which suggests we will need several more
> slaves (5-7?), lest we slow down the commit-bot. Do we have the resources for
> this?

I think we should scale down _valgrind and _layout slaves to replace them with
_clang. From slaves.cfg, we could add significantly more slaves to the mix. We
also added 8 more slaves last weekend.

[Nico, 2011-07-27 15:31:02]

On Wed, Jul 27, 2011 at 8:14 AM,  <maruel@chromium.org> wrote:
> On 2011/07/26 21:34:04, Nico wrote:
>>
>> This change increases load by 148%, which suggests we will need several
>> more
>> slaves (5-7?), lest we slow down the commit-bot. Do we have the resources
>> for
>> this?
>
> I think we should scale down _valgrind and _layout slaves to replace them
> with _clang. From slaves.cfg, we could add significantly more slaves to the
mix.
> We also added 8 more slaves last weekend.

It looks like both _valgrid and _layout trybots are handled by the
normal_slaves pool. Do you want to make that pool smaller? How many
bots do you think it can spare?

[M-A Ruel, 2011-07-27 15:50:59]

On 2011/07/27 15:31:02, Nico wrote:
> It looks like both _valgrid and _layout trybots are handled by the
> normal_slaves pool. Do you want to make that pool smaller? How many
> bots do you think it can spare?

Note that I'm skipping linux_chromeos_tsan because it didn't have a build since
last restart!

~/src/commit-queue> python2.7 buildbot_json.py interactive
build.chromium.org/p/tryserver.chromium
Buildbot interactive console for
"http://build.chromium.org/p/tryserver.chromium".
Hint: Start with typing: 'buildbot.printable_attributes' or 'print
str(buildbot)' to explore.
>>> import time
>>> builders = sorted([i for i in b.builders if i.name.startswith('linux') and
i.name != 'linux_chromeos_tsan'], key=lambda x:x.name)
>>> print '\n'.join(i.name for i in builders)
linux
linux_chromeos
linux_chromeos_clang
linux_chromeos_valgrind
linux_clang
linux_coverage
linux_layout
linux_layout_rel
linux_rel
linux_sync
linux_touch
linux_tsan
linux_valgrind
linux_view
>>> for builder in builders:
...   print '%s: %d' % (builder.name, len([i for i in builder.builds if
i.data['times'][0] > time.time() - 24*60*60]))
...
linux: 199
linux_chromeos: 63
linux_chromeos_clang: 23
linux_chromeos_valgrind: 5
linux_clang: 99
linux_coverage: 0
linux_layout: 4
linux_layout_rel: 2
linux_rel: 5
linux_sync: 17
linux_touch: 9
linux_tsan: 6
linux_valgrind: 18
linux_view: 44

So I plan to reshuffle the try slaves accordingly next weekend. Note that I plan
to enable _rel builders on the CQ eventually so I'll keep its number of slaves
high. Same for linux_view when it'll start using the preferred try builders.

[Timur Iskhodzhanov, 2011-07-27 16:03:01]

FTR, linux_chromeos_tsan bot uses the same checkout as linux_chromeos_valgrind;
the same for linux_tsan vs linux_valgrind

[Nico, 2011-07-27 16:08:48]

LGTM

worst case we can just revert this

On Wed, Jul 27, 2011 at 8:50 AM,  <maruel@chromium.org> wrote:
> On 2011/07/27 15:31:02, Nico wrote:
>>
>> It looks like both _valgrid and _layout trybots are handled by the
>> normal_slaves pool. Do you want to make that pool smaller? How many
>> bots do you think it can spare?
>
> Note that I'm skipping linux_chromeos_tsan because it didn't have a build
> since
> last restart!
>
> ~/src/commit-queue> python2.7 buildbot_json.py interactive
> build.chromium.org/p/tryserver.chromium
> Buildbot interactive console for
> "http://build.chromium.org/p/tryserver.chromium".
> Hint: Start with typing: 'buildbot.printable_attributes' or 'print
> str(buildbot)' to explore.
>>>>
>>>> import time
>>>> builders = sorted([i for i in b.builders if i.name.startswith('linux')
>>>> and
>
> i.name != 'linux_chromeos_tsan'], key=lambda x:x.name)
>>>>
>>>> print '\n'.join(i.name for i in builders)
>
> linux
> linux_chromeos
> linux_chromeos_clang
> linux_chromeos_valgrind
> linux_clang
> linux_coverage
> linux_layout
> linux_layout_rel
> linux_rel
> linux_sync
> linux_touch
> linux_tsan
> linux_valgrind
> linux_view
>>>>
>>>> for builder in builders:
>
> ...   print '%s: %d' % (builder.name, len([i for i in builder.builds if
> i.data['times'][0] > time.time() - 24*60*60]))
> ...
> linux: 199
> linux_chromeos: 63
> linux_chromeos_clang: 23
> linux_chromeos_valgrind: 5
> linux_clang: 99
> linux_coverage: 0
> linux_layout: 4
> linux_layout_rel: 2
> linux_rel: 5
> linux_sync: 17
> linux_touch: 9
> linux_tsan: 6
> linux_valgrind: 18
> linux_view: 44
>
> So I plan to reshuffle the try slaves accordingly next weekend. Note that I
> plan
> to enable _rel builders on the CQ eventually so I'll keep its number of
> slaves
> high. Same for linux_view when it'll start using the preferred try builders.
>
> http://codereview.chromium.org/7465043/
>

[M-A Ruel, 2011-07-27 16:13:51]

On 2011/07/27 16:03:01, Timur Iskhodzhanov wrote:
> FTR, linux_chromeos_tsan bot uses the same checkout as
linux_chromeos_valgrind;
> the same for linux_tsan vs linux_valgrind

FTR, slavebuilddir sharing goes this way:

linux, linux_rel, linux_sync
linux_tsan, linux_valgrind
linux_layout, linux_layout_rel

The _sync builders will need to die eventually.

[commit-bot: I haz the power, 2012-03-09 22:39:11]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/maruel@chromium.org/7465043/5001

[commit-bot: I haz the power, 2012-03-09 22:39:23]

Change committed as 125922