Add cpu_limit to ninja link pool on windows.

Add cpu_limit to ninja link pool on windows.

This will cap the number of links at the min of cpu_count v. mem / 4G.

R=scottmg@chromium.org

Committed: https://code.google.com/p/gyp/source/detail?r=1724

[scottmg, 2013-09-11 20:09:05]

https://codereview.chromium.org/24106005/diff/1/pylib/gyp/generator/ninja.py
File pylib/gyp/generator/ninja.py (right):

https://codereview.chromium.org/24106005/diff/1/pylib/gyp/generator/ninja.py#...
pylib/gyp/generator/ninja.py:1510: cpu_limit = multiprocessing.cpu_count()
that does

    if sys.platform == 'win32':
        try:
            num = int(os.environ['NUMBER_OF_PROCESSORS'])
        except (ValueError, KeyError):
            num = 0

so maybe clamp to >= 1 too?

[iannucci, 2013-09-11 20:36:21]

On 2013/09/11 20:09:05, scottmg wrote:
> https://codereview.chromium.org/24106005/diff/1/pylib/gyp/generator/ninja.py
> File pylib/gyp/generator/ninja.py (right):
> 
>
https://codereview.chromium.org/24106005/diff/1/pylib/gyp/generator/ninja.py#...
> pylib/gyp/generator/ninja.py:1510: cpu_limit = multiprocessing.cpu_count()
> that does
> 
>     if sys.platform == 'win32':
>         try:
>             num = int(os.environ['NUMBER_OF_PROCESSORS'])
>         except (ValueError, KeyError):
>             num = 0
> 
> so maybe clamp to >= 1 too?

Aw nuts. And here I was expecting something sane. I'd actually rather implement
it in terms of a win32 api call :/

[iannucci, 2013-09-12 00:18:58]

OK, PTAL (untested). I could also use some help testing this out if anyone of
you has a windows build machine handy (mine's a bit dusty).

I can test this on linux/mac.

[scottmg, 2013-09-12 00:24:38]

On 2013/09/12 00:18:58, iannucci wrote:
> OK, PTAL (untested). I could also use some help testing this out if anyone of
> you has a windows build machine handy (mine's a bit dusty).
> 
> I can test this on linux/mac.

Whoa, clamping to >= 1 seemed fine! It's highly unlikely that won't be set.

[iannucci, 2013-09-12 00:33:46]

On 2013/09/12 00:24:38, scottmg wrote:
> On 2013/09/12 00:18:58, iannucci wrote:
> > OK, PTAL (untested). I could also use some help testing this out if anyone
of
> > you has a windows build machine handy (mine's a bit dusty).
> > 
> > I can test this on linux/mac.
> 
> Whoa, clamping to >= 1 seemed fine! It's highly unlikely that won't be set.

fine!

*grumbles about environment variables*

(see PS #3)

[scottmg, 2013-09-12 00:34:11]

lgtm

[iannucci, 2013-09-12 00:39:50]

Committed patchset #4 manually as r1724 (presubmit successful).

[Nico, 2013-09-12 20:06:47]

When writing change descriptions either say why you're making a change, or point
to a bug that says why you are making a change. For this CL it's not clear to me
as linking often isn't cpu bound.

[iannucci, 2013-09-12 21:25:54]

On 2013/09/12 20:06:47, Nico wrote:
> When writing change descriptions either say why you're making a change, or
point
> to a bug that says why you are making a change. For this CL it's not clear to
me
> as linking often isn't cpu bound.

Ah, sorry. For posterity:

We have many bare-metal build configurations provisioned with a lot (128GB) of
memory, but with only a single CPU package (usually w/ 4 cores + HT). Because we
were only looking at memory usage previously, these machines ended up running
12+ simultaneous links which swamped the machine like crazy.

As you note, the link steps are not really CPU bound. Ideally we could model:
  * The IOPs supported by the underlying storage
  * The I/O profile of the link processes (which is dependent on things like
fastbuild, LTCG, etc.)
and then use this as the metric for limiting the links instead of cores.

We chose to use the core count as a rough approximation to strength of the
machine (since it was easy to calculate) to get the waterfalls unblocked in the
mean time. If we have a more accurate model of the way that the link processes
work, I think it would be easy to get the random-read latency of the underlying
storage, and then we could actually calculate a /really/ good number for this
concurrency.

That said, it looks like there's actually a variety of configurations in this
class of builder (some w/ 8 logical cores, some w/ 16+), and it's not really a
good heuristic anyway (for the reason you mention). I'm going to put up another
CL to reverse this one and hard cap at 4 (i.e. I'm 'giving up'). I'll put you on
the CL as well.