Cache data for included files in the multiprocess load codepath

Cache data for included files in the multiprocess load codepath

When running gyp with --no-parallel, the build file data for included gypi
files is cached in the 'data' dict, so we never load gypi files more than once.

However, when running in parallel mode, the |data| and |aux_data| parameters
passed to CallLoadTargetBuildFile are always empty.  That means gypi files are
reloaded for every single gyp file.

This commit changes CallLoadTargetBuildFile to use a global |per_process_data|
and |per_process_aux_data|, instead of accepting them as parameters.  This
allows each subprocess to get gypi file data from this per-process cache if it
has already encountered the gypi file when loading a previous target build
file.

Another minor change in this commit is to remove aux_data from ParallelState.
The aux_data is only ever used within the scope of LoadTargetBuildFile, so we
don't need to communicate it back to the main process.

BUG=
R=scottmg@chromium.org

Committed: https://code.google.com/p/gyp/source/detail?r=2012

[Shezan Baig (Bloomberg), 2014-12-02 19:28:37]

sbaig1@bloomberg.net changed reviewers:
+ scottmg@chromium.org

[Shezan Baig (Bloomberg), 2014-12-02 19:28:38]

Scott, I noticed that you have done some work with the parallel load stuff, so I
hope you're the right person to review this :)

I tried this patch in chromium and it improves gyp time by about a second on my
Windows machine.

[scottmg, 2014-12-02 22:50:53]

lgtm

https://codereview.chromium.org/773883002/diff/1/pylib/gyp/input.py
File pylib/gyp/input.py (right):

https://codereview.chromium.org/773883002/diff/1/pylib/gyp/input.py#newcode466
pylib/gyp/input.py:466: per_process_data = {}
please move these up to the top

https://codereview.chromium.org/773883002/diff/1/pylib/gyp/input.py#newcode477
pylib/gyp/input.py:477: 
can you add

  global per_process_data
  global per_process_aux_data

here (I know it's not necessary, but just to make it a bit more obvious)

https://codereview.chromium.org/773883002/diff/1/pylib/gyp/input.py#newcode497
pylib/gyp/input.py:497: build_file_data = per_process_data[build_file_path]
build_file_data = per_process_data.pop(build_file_path)

[Shezan Baig (Bloomberg), 2014-12-03 14:24:16]

PTAL - I removed the unused 'aux_data' parameter in LoadTargetBuildFilesParallel

Also, pylint complains about:

    W:480, 2: Using global for 'per_process_data' but no assignment is done
(global-variable-not-assigned)
    W:481, 2: Using global for 'per_process_aux_data' but no assignment is done
(global-variable-not-assigned)

Should I just remove those global statements, or disable this pylint warning?

[scottmg, 2014-12-03 17:41:34]

On 2014/12/03 14:24:16, Shezan Baig (Bloomberg) wrote:
> PTAL - I removed the unused 'aux_data' parameter in
LoadTargetBuildFilesParallel
> 
> Also, pylint complains about:
> 
>     W:480, 2: Using global for 'per_process_data' but no assignment is done
> (global-variable-not-assigned)
>     W:481, 2: Using global for 'per_process_aux_data' but no assignment is
done
> (global-variable-not-assigned)
> 
> Should I just remove those global statements, or disable this pylint warning?

Ah ok, sorry. I'm finding pylint to be a "Great Asset" :p. But ignore me then,
and remove them. :)

Still lgtm.

[Shezan Baig (Bloomberg), 2014-12-03 17:51:09]

On 2014/12/03 17:41:34, scottmg wrote:
> On 2014/12/03 14:24:16, Shezan Baig (Bloomberg) wrote:
> > PTAL - I removed the unused 'aux_data' parameter in
> LoadTargetBuildFilesParallel
> > 
> > Also, pylint complains about:
> > 
> >     W:480, 2: Using global for 'per_process_data' but no assignment is done
> > (global-variable-not-assigned)
> >     W:481, 2: Using global for 'per_process_aux_data' but no assignment is
> done
> > (global-variable-not-assigned)
> > 
> > Should I just remove those global statements, or disable this pylint
warning?
> 
> Ah ok, sorry. I'm finding pylint to be a "Great Asset" :p. But ignore me then,
> and remove them. :)
> 
> Still lgtm.

K, thanks :)

[Shezan Baig (Bloomberg), 2014-12-03 18:46:11]

Committed patchset #4 (id:60001) manually as 2012 (presubmit successful).