Implement multi-core builds in MSVS

Implement multi-core builds.

This is done using a cl.exe compatability wrapper around
gcc that supports multiple sources files and builds them
in parallel.


BUG=

Committed: https://code.google.com/p/nativeclient-sdk/source/detail?r=1430

[Sam Clegg, 2012-10-03 22:09:57]

Initial version of the muilticore builder.  Not yet
ready for submit...

[binji, 2012-10-03 22:52:13]

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:1:
#!/usr/bin/env python
add license header

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:9:
import time
sort imports alphabetically

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:22:
if show_commands:
nit: move down to use of logmsg

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:33:
def BuildSerial(base_cmd, outpath, files):
Is this necessary? Why not always use BuildParallel below?

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:49:
q = Queue.Queue()
IMO, it is nicer to make these local to BuildParallel and pass them via args in
threading.Thread below.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:54:
while not q.empty():
and Worker.running?

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:64:
out = os.path.join(outpath, os.path.splitext(basename)[0] + '.obj')
use .o instead of .obj?

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:86:
sys.stderr.write(stdout)
why write stdout from compiler to stderr?

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:114:
# find the last occurance of '--' in the argument
nit: s/occurance/occurrence/

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:115:
# list an use that to signify the start of the
s/an/and/

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:140:
Trace("done build of %d sources" % (len(files)))
what about when the build fails?

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:110: if
((File.GetAttributes(objectFilePath) & FileAttributes.Directory) != 0)
might be nice to have a comment here explaining why this is necessary

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:299:
Dictionary<string, List<ITaskItem>> srcGroups = new Dictionary<string,
List<ITaskItem>>();
nit: wrap line

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:313: string
pythonExe = "C:\\python_26_amd64\\files\\python.exe";
Probably need a better way to find the python exe.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:325: string
sourcePath = GCCUtilities.Convert_Path_Windows_To_Posix(sourceItem.ToString());
nit: wrap line

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/XamlParser.cs
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/XamlParser.cs:134:
//AppendStringValue(builder, property, String.Empty, value);
remove unused code

[Sam Clegg, 2012-10-04 22:57:57]

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:1:
#!/usr/bin/env python
On 2012/10/03 22:52:13, binji wrote:
> add license header

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:9:
import time
On 2012/10/03 22:52:13, binji wrote:
> sort imports alphabetically

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:22:
if show_commands:
On 2012/10/03 22:52:13, binji wrote:
> nit: move down to use of logmsg

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:33:
def BuildSerial(base_cmd, outpath, files):
On 2012/10/03 22:52:13, binji wrote:
> Is this necessary? Why not always use BuildParallel below?

Just here for debugging.  This code path is not used.
Should I remove it?

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:49:
q = Queue.Queue()
On 2012/10/03 22:52:13, binji wrote:
> IMO, it is nicer to make these local to BuildParallel and pass them via args
in
> threading.Thread below.

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:54:
while not q.empty():
On 2012/10/03 22:52:13, binji wrote:
> and Worker.running?

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:64:
out = os.path.join(outpath, os.path.splitext(basename)[0] + '.obj')
On 2012/10/03 22:52:13, binji wrote:
> use .o instead of .obj?

Unfortunately we can't in this case as we need the OutFile
to be the same for each object.

The way visual studio does this is to allow you to specify
a directory as the '-o' file in which case the object are
supposed to be created as "OutFile/<basename>.obj".

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:86:
sys.stderr.write(stdout)
On 2012/10/03 22:52:13, binji wrote:
> why write stdout from compiler to stderr?

Good question.  Visual studio doesn't display stdout
from cl.exe it seems.  Only shows stderr.  Do we case if
GCC stdout is lost?  I imagine all the important stuff is
on stderr already right?   As it stands I link we are already loosing stdout.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:114:
# find the last occurance of '--' in the argument
On 2012/10/03 22:52:13, binji wrote:
> nit: s/occurance/occurrence/

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:115:
# list an use that to signify the start of the
On 2012/10/03 22:52:13, binji wrote:
> s/an/and/

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:140:
Trace("done build of %d sources" % (len(files)))
On 2012/10/03 22:52:13, binji wrote:
> what about when the build fails?

Well spotted.  Amazingly (you are not going to believe this)
visual studio seems to look at the error messages on stdout
to decide if the build failed and not at the return code.

I will fix this anyway... and double confirm what I am seeing.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:110: if
((File.GetAttributes(objectFilePath) & FileAttributes.Directory) != 0)
On 2012/10/03 22:52:13, binji wrote:
> might be nice to have a comment here explaining why this is necessary

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:299:
Dictionary<string, List<ITaskItem>> srcGroups = new Dictionary<string,
List<ITaskItem>>();
On 2012/10/03 22:52:13, binji wrote:
> nit: wrap line

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:313: string
pythonExe = "C:\\python_26_amd64\\files\\python.exe";
On 2012/10/03 22:52:13, binji wrote:
> Probably need a better way to find the python exe.

Yup.. will be sure to find a better way before I land this.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/NaClCompile.cs:325: string
sourcePath = GCCUtilities.Convert_Path_Windows_To_Posix(sourceItem.ToString());
On 2012/10/03 22:52:13, binji wrote:
> nit: wrap line

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/XamlParser.cs
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/NaCl.Build.CPPTasks/XamlParser.cs:134:
//AppendStringValue(builder, property, String.Empty, value);
On 2012/10/03 22:52:13, binji wrote:
> remove unused code

Done.

[binji, 2012-10-04 23:27:31]

lgtm

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:33:
def BuildSerial(base_cmd, outpath, files):
On 2012/10/04 22:57:57, Sam Clegg wrote:
> On 2012/10/03 22:52:13, binji wrote:
> > Is this necessary? Why not always use BuildParallel below?
> 
> Just here for debugging.  This code path is not used.
> Should I remove it?

Looks like it will still run if cores==1... but it seems like there is enough
duplicated logic that it would make sense to remove. Your call.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:64:
out = os.path.join(outpath, os.path.splitext(basename)[0] + '.obj')
On 2012/10/04 22:57:57, Sam Clegg wrote:
> On 2012/10/03 22:52:13, binji wrote:
> > use .o instead of .obj?
> 
> Unfortunately we can't in this case as we need the OutFile
> to be the same for each object.
> 
> The way visual studio does this is to allow you to specify
> a directory as the '-o' file in which case the object are
> supposed to be created as "OutFile/<basename>.obj".

ok, comment would be nice. :)

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:86:
sys.stderr.write(stdout)
On 2012/10/04 22:57:57, Sam Clegg wrote:
> On 2012/10/03 22:52:13, binji wrote:
> > why write stdout from compiler to stderr?
> 
> Good question.  Visual studio doesn't display stdout
> from cl.exe it seems.  Only shows stderr.  Do we case if
> GCC stdout is lost?  I imagine all the important stuff is
> on stderr already right?   As it stands I link we are already loosing stdout.

OK, might add a comment then.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:140:
Trace("done build of %d sources" % (len(files)))
On 2012/10/04 22:57:57, Sam Clegg wrote:
> On 2012/10/03 22:52:13, binji wrote:
> > what about when the build fails?
> 
> Well spotted.  Amazingly (you are not going to believe this)
> visual studio seems to look at the error messages on stdout
> to decide if the build failed and not at the return code.
> 
> I will fix this anyway... and double confirm what I am seeing.

Haha, I actually meant the Trace is displaying that the build finished the
sources, even though it may not have.

http://codereview.chromium.org/11044024/diff/6001/visual_studio/NativeClientV...
File
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py
(right):

http://codereview.chromium.org/11044024/diff/6001/visual_studio/NativeClientV...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:152:
main(sys.argv[1:])
sys.exit(...) to send the errorcode (as long as you made the change above)

[Sam Clegg, 2012-10-05 21:11:19]

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
File
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py
(right):

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:86:
sys.stderr.write(stdout)
On 2012/10/04 23:27:31, binji wrote:
> On 2012/10/04 22:57:57, Sam Clegg wrote:
> > On 2012/10/03 22:52:13, binji wrote:
> > > why write stdout from compiler to stderr?
> > 
> > Good question.  Visual studio doesn't display stdout
> > from cl.exe it seems.  Only shows stderr.  Do we case if
> > GCC stdout is lost?  I imagine all the important stuff is
> > on stderr already right?   As it stands I link we are already loosing
stdout.
> 
> OK, might add a comment then.

Done.

http://codereview.chromium.org/11044024/diff/1/visual_studio/NativeClientVSAd...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:140:
Trace("done build of %d sources" % (len(files)))
On 2012/10/04 23:27:31, binji wrote:
> On 2012/10/04 22:57:57, Sam Clegg wrote:
> > On 2012/10/03 22:52:13, binji wrote:
> > > what about when the build fails?
> > 
> > Well spotted.  Amazingly (you are not going to believe this)
> > visual studio seems to look at the error messages on stdout
> > to decide if the build failed and not at the return code.
> > 
> > I will fix this anyway... and double confirm what I am seeing.
> 
> Haha, I actually meant the Trace is displaying that the build finished the
> sources, even though it may not have.

Done.

http://codereview.chromium.org/11044024/diff/6001/visual_studio/NativeClientV...
File
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py
(right):

http://codereview.chromium.org/11044024/diff/6001/visual_studio/NativeClientV...
visual_studio/NativeClientVSAddIn/InstallerResources/NaCl/compiler_wrapper.py:152:
main(sys.argv[1:])
On 2012/10/04 23:27:31, binji wrote:
> sys.exit(...) to send the errorcode (as long as you made the change above)

Done.