Build system for statically-linked Python.

Build system for statically-linked Python.

This adds a new subdirectory, python_modules/ , and a mechanism for
creating a statically linked Python build.  It is designed primarily for
PNaCl, where a lack of shared library support and the cost of
translation mean that a large Python stack requires a complex,
multi-step build process.  This process can be summarized as such:

 * Build a host python containing all necessary build-time dependencies
   for the stack
 * Build each individual module
 * Collect object files and construct unified libpython2.7.a and
   python.pexe
 * Add a dynload module for statically linked modules

When not automated, this is prone to error and difficult to add or
remove additional modules.  The system included here, while not perfect,
addresses this by enabling individual modules to be selected and built
ala carte, with a reubilding of python_static necessary to create the
unified python.pexe and libpython2.7.a.  The libpython2.7.a output by
this can then be used as a link target for a python_ppapi-like module.

As examples, and as components of a scientific python stack, this
changelist includes:

 * numpy (host and destination builds)
 * cython (host-only build)
 * setuptools (host-only build)
 * matplotlib (dest-only build)
 * sympy (dest-only build)
 * ipython (dest-only build)
 * pyzmq (dest-only build)
 * h5py (dest-only build; requires hdf5, under separate CL review)
 * jinja2 (dest-only build)
 * pyppapi (dest-only build)
 * six (dest-only build)
 * distribute (dest-only build)
 * nose (dest-only build)
 * pyparsing (dest-only build)
 * dateutil (dest-only build)
 * markupsafe (dest-only build)
 * yt (dest-only build)

This CL includes a port for the pyppapi module, which was developed in
support of this and which provides a Python API to communicate between
running Native Client modules and Chrome.  This includes serialization
of NumPy arrays into typed arrays as well as other Python/JS
translations.

Python module ports act very similarly to standard naclports modules,
with two exceptions.  The installation of the .py files has been moved
into the building phase and the collection of .o files occurs during
installation.  Additionally, they utilize modules.libs and modules.list
files.  The first of these files should append to the variable
PY_MOD_LINK to add on additional libraries for building the module, and
the second is a list of compiled modules (not object files) produced by
the python module which should be included in the symbol table in
libpython2.7.a.  For instance, if the compiled C module "fastcode"
exists in a Python module and it requires linking against cfast, these
files would look like:

modules.list:

fastcode

 modules.libs:

PY_MOD_LINK+=" -lcfast"

At present, this has been tested on PNaCl.  To construct a Python build
with a number of modules, a command like this can be issues:

NACL_ARCH=pnacl make numpy sympy matplotlib python_static

The resultant installed libpython2.7.a will contain all of those
modules.  Note that any pure-python modules auto-installed by setuptools
will be functional in the resultant python destination, but any
C-compiled python modules that have been auto-installed will simply
fail.

R=sbc@chromium.org, hamaji@chromium.org, kestert@google.com
BUG=

Committed: https://code.google.com/p/naclports/source/detail?r=1230

[Matthew Turk, 2014-01-20 15:22:20]

This changelist includes a number of new additions, primarily in the way of
adding the ability to compile in python modules for a large statically-linked
Python.  The goal is to be able to create a python that includes the full
scientific stack, and here it has been constructed to include IPython,
matplotlib, numpy, yt, h5py, and sympy.  This isn't a high-priority commit and
I'd be fine with review / suggestions taking a while and undergoing
modifications, or even being outright declined if it's too big or mismatched for
naclports.

Thanks for any suggestions or comments!

[Sam Clegg, 2014-01-22 01:46:49]

This change looks really promising.   I've not had time to look at it in depth
yet, and I don't have much sci-py knowledge  but it looks like a reasonable
solution to the problem to me.

https://codereview.chromium.org/138913004/diff/1/build_tools/common.sh
File build_tools/common.sh (right):

https://codereview.chromium.org/138913004/diff/1/build_tools/common.sh#newcod...
build_tools/common.sh:871: if ! CheckStamp install_dest_${PACKAGE_NAME} ; then
Maybe early return here rather than indent to whole function?

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython...
File ports/python_modules/ipython/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython...
ports/python_modules/ipython/nacl.patch:35: -devnull = open(os.devnull, 'a')
Didn't you recently fix this in nacl_io?   Perhaps we can avoid
this by depending on a very recent version of the SDK.

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/numpy/n...
File ports/python_modules/numpy/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/numpy/n...
ports/python_modules/numpy/nacl.patch:99: +        if
os.environ.get("NACL_PORT_BUILD", None) == "bootstrap":
nit: you can remove the second arg here as None is the default.

[Matthew Turk, 2014-01-22 21:14:31]

Thanks for the comments, Sam!  I've updated with those, and removed a few
unrelated changes.  No rush on any followup suggestions, as I'm hoping to get
pyppapi stable, released, documented and a nice example site up before this CL
goes in.

https://codereview.chromium.org/138913004/diff/1/build_tools/common.sh
File build_tools/common.sh (right):

https://codereview.chromium.org/138913004/diff/1/build_tools/common.sh#newcod...
build_tools/common.sh:871: if ! CheckStamp install_dest_${PACKAGE_NAME} ; then
On 2014/01/22 01:46:50, Sam Clegg wrote:
> Maybe early return here rather than indent to whole function?

Done.

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython...
File ports/python_modules/ipython/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython...
ports/python_modules/ipython/nacl.patch:35: -devnull = open(os.devnull, 'a')
On 2014/01/22 01:46:50, Sam Clegg wrote:
> Didn't you recently fix this in nacl_io?   Perhaps we can avoid
> this by depending on a very recent version of the SDK.

Good catch.  I've removed the chunk of the patch, but don't know how to mandate
the SDK except by assumption.

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/numpy/n...
File ports/python_modules/numpy/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/numpy/n...
ports/python_modules/numpy/nacl.patch:99: +        if
os.environ.get("NACL_PORT_BUILD", None) == "bootstrap":
On 2014/01/22 01:46:50, Sam Clegg wrote:
> nit: you can remove the second arg here as None is the default.

Done.

[Sam Clegg, 2014-01-22 22:22:26]

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython...
File ports/python_modules/ipython/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython...
ports/python_modules/ipython/nacl.patch:35: -devnull = open(os.devnull, 'a')
On 2014/01/22 21:14:31, Matthew Turk wrote:
> On 2014/01/22 01:46:50, Sam Clegg wrote:
> > Didn't you recently fix this in nacl_io?   Perhaps we can avoid
> > this by depending on a very recent version of the SDK.
> 
> Good catch.  I've removed the chunk of the patch, but don't know how to
mandate
> the SDK except by assumption.

I think you can do MIN_SDK_VERSION=34 in pkg_info

[hamaji, 2014-01-24 14:32:48]

Please note that I don't think I'm a good reviewer for naclports changes because
I'm not familiar with naclports. I'm just a random contributor for naclports and
am not working in NaCl team. Sorry for the latency of this response.

Anyway, I've put some nitpicks. If Sam has a different opinion or you think my
comment is not appropriate, please feel free to disregard my comment.

https://codereview.chromium.org/138913004/diff/830002/build_tools/common.sh
File build_tools/common.sh (right):

https://codereview.chromium.org/138913004/diff/830002/build_tools/common.sh#n...
build_tools/common.sh:912: Banner "Installing"
"Installing ${PACKAGE_NAME}" ?

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/h5...
File ports/python_modules/h5py/build.sh (right):

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/h5...
ports/python_modules/h5py/build.sh:6: NACL_PYSETUP_ARGS=""
Do we need this?

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/nu...
File ports/python_modules/numpy/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/nu...
ports/python_modules/numpy/nacl.patch:37: +#elif defined(__x86_64__) ||
defined(__amd64__) || defined(__x86_64) || defined(_M_AMD64) || defined
(__native_client__)
Maybe better to explain why we need this change.

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/nu...
ports/python_modules/numpy/nacl.patch:112: +#if (IEEE_WORD_ORDER ==
NPY_LITTLE_ENDIAN) || defined(__native_client__)
Why have you swapped the order of this #if clause? I assume IEEE_WORD_ORDER will
be NPY_BIG_ENDIAN for some reason? If so, cannot we set a correct value to
IEEE_WORD_ORDER? For example, adding something like

#if defined(__native_client__) && (defined(__i686__) || defined(__x86_64__)
#undef IEEE_WORD_ORDER
#define IEEE_WORD_ORDER NPY_LITTLE_ENDIAN
#endif

won't work?

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/py...
File ports/python_modules/pyzmq/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/py...
ports/python_modules/pyzmq/nacl.patch:168: +
Why this file need to be added? Cannot we add a comment about this?

This makes the diff much harder to read. Cannot we check in these added files in
ports directory and copy them in the patch step? I don't know the good practice
of naclports, but I've seen similar script in other ports (e.g., bash_pepper.c)

Ah, you've just renamed some files and modified them a bit? I guess you can use
git -M to create this file. This would reduce the size of diffs significantly.

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/py...
ports/python_modules/pyzmq/nacl.patch:260: +
ditto. Why added?

[Matthew Turk, 2014-01-28 19:39:19]

Thanks for the feedback.  I've gone through and addressed the comments, except
for the pyzmq/nacl.patch, which I'm not sure yet how to apply since using
renames requires extended diff format.

https://codereview.chromium.org/138913004/diff/830002/build_tools/common.sh
File build_tools/common.sh (right):

https://codereview.chromium.org/138913004/diff/830002/build_tools/common.sh#n...
build_tools/common.sh:912: Banner "Installing"
On 2014/01/24 14:32:48, hamaji wrote:
> "Installing ${PACKAGE_NAME}" ?

Done.

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/h5...
File ports/python_modules/h5py/build.sh (right):

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/h5...
ports/python_modules/h5py/build.sh:6: NACL_PYSETUP_ARGS=""
On 2014/01/24 14:32:48, hamaji wrote:
> Do we need this?

Done.

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/nu...
File ports/python_modules/numpy/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/nu...
ports/python_modules/numpy/nacl.patch:37: +#elif defined(__x86_64__) ||
defined(__amd64__) || defined(__x86_64) || defined(_M_AMD64) || defined
(__native_client__)
On 2014/01/24 14:32:48, hamaji wrote:
> Maybe better to explain why we need this change.

Done.

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/nu...
ports/python_modules/numpy/nacl.patch:112: +#if (IEEE_WORD_ORDER ==
NPY_LITTLE_ENDIAN) || defined(__native_client__)
On 2014/01/24 14:32:48, hamaji wrote:
> Why have you swapped the order of this #if clause? I assume IEEE_WORD_ORDER
will
> be NPY_BIG_ENDIAN for some reason? If so, cannot we set a correct value to
> IEEE_WORD_ORDER? For example, adding something like
> 
> #if defined(__native_client__) && (defined(__i686__) || defined(__x86_64__)
> #undef IEEE_WORD_ORDER
> #define IEEE_WORD_ORDER NPY_LITTLE_ENDIAN
> #endif
> 
> won't work?

It might.  The issue I was running into is that for the PNaCl toolchain, the
criteria it was using to evaluate IEEE_WORD_ORDER was giving truth values for
both little *and* big endian, and the PNaCl target is little endian, so I needed
to break the symmetry somehow.  In the current version of the patch I have not
changed this, but I will evaluate for a subsequent version.  I would prefer a
solution like yours.

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/py...
File ports/python_modules/pyzmq/nacl.patch (right):

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/py...
ports/python_modules/pyzmq/nacl.patch:168: +
On 2014/01/24 14:32:48, hamaji wrote:
> Why this file need to be added? Cannot we add a comment about this?
> 
> This makes the diff much harder to read. Cannot we check in these added files
in
> ports directory and copy them in the patch step? I don't know the good
practice
> of naclports, but I've seen similar script in other ports (e.g.,
bash_pepper.c)
> 
> Ah, you've just renamed some files and modified them a bit? I guess you can
use
> git -M to create this file. This would reduce the size of diffs significantly.

Thanks for the tip, didn't know about it.  But it looks like the actual patching
is done via patch, not git, so I'm not sure it will work.  I've left the
nacl.patch as is, although I have one created with "git diff -M upstream" that
I'd be happy to swap in.  Sam, do you have any suggestions on this?  I could
also do the renaming step inside the build.sh.

https://codereview.chromium.org/138913004/diff/830002/ports/python_modules/py...
ports/python_modules/pyzmq/nacl.patch:260: +
On 2014/01/24 14:32:48, hamaji wrote:
> ditto. Why added?

Done.

[Kester Tong, 2014-01-29 20:38:47]

Hey,

This looks really cool, thanks for this contribution.  I've been trying to build
this CL on my machine, and I found a few places where I needed to make changes
to get it to build.  See the comments for my suggested fixes.

Kester

https://codereview.chromium.org/138913004/diff/1130001/Makefile
File Makefile (right):

https://codereview.chromium.org/138913004/diff/1130001/Makefile#newcode306
Makefile:306: ports/python_modules/pyppapi ports/openssl
does cython belong here too?

https://codereview.chromium.org/138913004/diff/1130001/ports/python_modules/n...
File ports/python_modules/numpy/build.sh (right):

https://codereview.chromium.org/138913004/diff/1130001/ports/python_modules/n...
ports/python_modules/numpy/build.sh:17: export CC="gcc -m32"
On my machine, the -m32 flag disables SSE instructions, and since this build
uses some system headers that, on my machine, contain inline assembly with SSE
instructions, this will not compile.

Can you add the -msse flag to these three lines, which manually re-enables these
SSE instructions so that they can compile on systems such as mine?

https://codereview.chromium.org/138913004/diff/1130001/ports/python_static/bu...
File ports/python_static/build.sh (right):

https://codereview.chromium.org/138913004/diff/1130001/ports/python_static/bu...
ports/python_static/build.sh:42: source ${MODFILE}
in the case that the glob ${DEST_PYTHON_OBJS}/*.libs has zero matches, bash will
call source ${DEST_PYTHON_OBJS}/*.libs resulting in an error.  The easiest fix
for this is to check that MODFILE exists, so that

    source ${MODFILE}

is replaced by

    if [ -e ${MODFILE} ]; then
      source ${MODFILE}
    fi

[Matthew Turk, 2014-01-31 15:56:56]

Thanks for the kind words and comments, Kester!  I've updated with those changes
now.  I'm currently stuck on this bug:

https://code.google.com/p/nativeclient/issues/detail?id=3779

which is preventing me from doing some additional testing, but I was able to
build the system with the modifications.  Also, if anyone has any ideas how to
get the pyzmq patch down in size and use renames, I'd be eager to hear them. 
I'm going to take another shot at modifying the setup.py files to build from one
pyx file into another differently-named module for the next revision.

https://codereview.chromium.org/138913004/diff/1130001/Makefile
File Makefile (right):

https://codereview.chromium.org/138913004/diff/1130001/Makefile#newcode306
Makefile:306: ports/python_modules/pyppapi ports/openssl
On 2014/01/29 20:38:47, Kester Tong wrote:
> does cython belong here too?

It's needed, but only for pyppapi -- so I've updated and put it under the
pyppapi dependency.

https://codereview.chromium.org/138913004/diff/1130001/ports/python_modules/n...
File ports/python_modules/numpy/build.sh (right):

https://codereview.chromium.org/138913004/diff/1130001/ports/python_modules/n...
ports/python_modules/numpy/build.sh:17: export CC="gcc -m32"
On 2014/01/29 20:38:47, Kester Tong wrote:
> On my machine, the -m32 flag disables SSE instructions, and since this build
> uses some system headers that, on my machine, contain inline assembly with SSE
> instructions, this will not compile.
> 
> Can you add the -msse flag to these three lines, which manually re-enables
these
> SSE instructions so that they can compile on systems such as mine?

Done.

https://codereview.chromium.org/138913004/diff/1130001/ports/python_static/bu...
File ports/python_static/build.sh (right):

https://codereview.chromium.org/138913004/diff/1130001/ports/python_static/bu...
ports/python_static/build.sh:42: source ${MODFILE}
On 2014/01/29 20:38:47, Kester Tong wrote:
> in the case that the glob ${DEST_PYTHON_OBJS}/*.libs has zero matches, bash
will
> call source ${DEST_PYTHON_OBJS}/*.libs resulting in an error.  The easiest fix
> for this is to check that MODFILE exists, so that
> 
>     source ${MODFILE}
> 
> is replaced by
> 
>     if [ -e ${MODFILE} ]; then
>       source ${MODFILE}
>     fi

Done.

[Kester Tong, 2014-02-05 18:08:03]

Thanks for making those changes.  I'm not sure about renaming files
in a patch file either.

I was also wondering if your implementation of pyzmq/zmq was complete
(including the Chrome side)?  If this is proving to be a difficulty,
I wanted to point out a different way to create an IPython kernel for
IPython notebook.

Right now, the IPython zmq kernel is a wrapper around the IPython
interactive shell (e.g. see zmqshell.py).  It is also possible to
create a different wrapper than uses pepper messaging in place of
zmq.  This would involve creating suitably modified versions of
ipkernel.py, kernelapp.py, and zmqshell.py. It is then easy to
modify kernel.js to receive these pepper messages.

This approach allows for an IPython kernel that you can connect to
from javascript, that doesn't require zmq to be implemented.

On 2014/01/31 15:56:56, Matthew Turk wrote:
> Thanks for the kind words and comments, Kester!  I've updated with those
changes
> now.  I'm currently stuck on this bug:
> 
> https://code.google.com/p/nativeclient/issues/detail?id=3779
> 
> which is preventing me from doing some additional testing, but I was able to
> build the system with the modifications.  Also, if anyone has any ideas how to
> get the pyzmq patch down in size and use renames, I'd be eager to hear them. 
> I'm going to take another shot at modifying the setup.py files to build from
one
> pyx file into another differently-named module for the next revision.
> 
> https://codereview.chromium.org/138913004/diff/1130001/Makefile
> File Makefile (right):
> 
> https://codereview.chromium.org/138913004/diff/1130001/Makefile#newcode306
> Makefile:306: ports/python_modules/pyppapi ports/openssl
> On 2014/01/29 20:38:47, Kester Tong wrote:
> > does cython belong here too?
> 
> It's needed, but only for pyppapi -- so I've updated and put it under the
> pyppapi dependency.
> 
>
https://codereview.chromium.org/138913004/diff/1130001/ports/python_modules/n...
> File ports/python_modules/numpy/build.sh (right):
> 
>
https://codereview.chromium.org/138913004/diff/1130001/ports/python_modules/n...
> ports/python_modules/numpy/build.sh:17: export CC="gcc -m32"
> On 2014/01/29 20:38:47, Kester Tong wrote:
> > On my machine, the -m32 flag disables SSE instructions, and since this build
> > uses some system headers that, on my machine, contain inline assembly with
SSE
> > instructions, this will not compile.
> > 
> > Can you add the -msse flag to these three lines, which manually re-enables
> these
> > SSE instructions so that they can compile on systems such as mine?
> 
> Done.
> 
>
https://codereview.chromium.org/138913004/diff/1130001/ports/python_static/bu...
> File ports/python_static/build.sh (right):
> 
>
https://codereview.chromium.org/138913004/diff/1130001/ports/python_static/bu...
> ports/python_static/build.sh:42: source ${MODFILE}
> On 2014/01/29 20:38:47, Kester Tong wrote:
> > in the case that the glob ${DEST_PYTHON_OBJS}/*.libs has zero matches, bash
> will
> > call source ${DEST_PYTHON_OBJS}/*.libs resulting in an error.  The easiest
fix
> > for this is to check that MODFILE exists, so that
> > 
> >     source ${MODFILE}
> > 
> > is replaced by
> > 
> >     if [ -e ${MODFILE} ]; then
> >       source ${MODFILE}
> >     fi
> 
> Done.

[Matthew Turk, 2014-02-08 02:57:07]

This patchset updates to match the new build system, and includes a modification
to enable the python_modules/ subdirectory to be part of gen.mk.  Kester and I
talked a bit, and a separate CL looks like the best way to handle additional
IPython kernel work.

[Matthew Turk, 2014-03-03 19:50:02]

This patch addresses all previous comments, applies against the current svn
HEAD, and avoids the need to have client-side libssl for the python_host build,
as all dependencies are included throughout the stack.  I believe it may be
ready to go.

[Matthew Turk, 2014-03-03 20:05:03]

On 2014/03/03 19:50:02, Matthew Turk wrote:
> This patch addresses all previous comments, applies against the current svn
> HEAD, and avoids the need to have client-side libssl for the python_host
build,
> as all dependencies are included throughout the stack.  I believe it may be
> ready to go.

Apologies, this still needs work for SHA1 to be put in pkg_info.  I'm now fixing
that.

[Matthew Turk, 2014-03-04 01:30:51]

Sorry for the false alarm earlier -- this updates everything to the new build
system, and I have tested with "NACL_ARCH=pnacl make yt python_static" from
clean pepper_canary and naclports systems.

ptal

Thanks for any feedback, but I am getting pretty happy with where this is right
now.

[Matthew Turk, 2014-04-07 17:38:01]

On 2014/03/04 01:30:51, Matthew Turk wrote:
> Sorry for the false alarm earlier -- this updates everything to the new build
> system, and I have tested with "NACL_ARCH=pnacl make yt python_static" from
> clean pepper_canary and naclports systems.
> 
> ptal
> 
> Thanks for any feedback, but I am getting pretty happy with where this is
right
> now.

Hi all, any chance someone could take a look at this, and pending any comments
I'll commit it?

[Sam Clegg, 2014-04-07 18:25:32]

On 2014/04/07 17:38:01, Matthew Turk wrote:
> On 2014/03/04 01:30:51, Matthew Turk wrote:
> > Sorry for the false alarm earlier -- this updates everything to the new
build
> > system, and I have tested with "NACL_ARCH=pnacl make yt python_static" from
> > clean pepper_canary and naclports systems.
> > 
> > ptal
> > 
> > Thanks for any feedback, but I am getting pretty happy with where this is
> right
> > now.
> 
> Hi all, any chance someone could take a look at this, and pending any comments
> I'll commit it?

Would you mind re-basing onto current HEAD.  I tried to use the patch but it
doesn't currently apply.

I just wanted to make sure the all the binaries are on the mirror before we land
this but the this looks good to me otherwise.

[Matthew Turk, 2014-04-08 13:19:26]

On 2014/04/07 18:25:32, Sam Clegg wrote:
> On 2014/04/07 17:38:01, Matthew Turk wrote:
> > On 2014/03/04 01:30:51, Matthew Turk wrote:
> > > Sorry for the false alarm earlier -- this updates everything to the new
> build
> > > system, and I have tested with "NACL_ARCH=pnacl make yt python_static"
from
> > > clean pepper_canary and naclports systems.
> > > 
> > > ptal
> > > 
> > > Thanks for any feedback, but I am getting pretty happy with where this is
> > right
> > > now.
> > 
> > Hi all, any chance someone could take a look at this, and pending any
comments
> > I'll commit it?
> 
> Would you mind re-basing onto current HEAD.  I tried to use the patch but it
> doesn't currently apply.
> 
> I just wanted to make sure the all the binaries are on the mirror before we
land
> this but the this looks good to me otherwise.

Hi Sam, I've been working this morning to update the python_modules subdirectory
to work with the new build system, and I'm running into an odd problem.  One of
the modules, python-host (formerly python_host) shouldn't be installing
anywhere, and so during the PAYLOAD inspection it's failing as it finds all the
non-normalized posix paths.  Is it possible to disable the installation step
somehow?  (The specific error is "Invalid Package file")

[Sam Clegg, 2014-04-08 15:38:30]

On 2014/04/08 13:19:26, Matthew Turk wrote:
> On 2014/04/07 18:25:32, Sam Clegg wrote:
> > On 2014/04/07 17:38:01, Matthew Turk wrote:
> > > On 2014/03/04 01:30:51, Matthew Turk wrote:
> > > > Sorry for the false alarm earlier -- this updates everything to the new
> > build
> > > > system, and I have tested with "NACL_ARCH=pnacl make yt python_static"
> from
> > > > clean pepper_canary and naclports systems.
> > > > 
> > > > ptal
> > > > 
> > > > Thanks for any feedback, but I am getting pretty happy with where this
is
> > > right
> > > > now.
> > > 
> > > Hi all, any chance someone could take a look at this, and pending any
> comments
> > > I'll commit it?
> > 
> > Would you mind re-basing onto current HEAD.  I tried to use the patch but it
> > doesn't currently apply.
> > 
> > I just wanted to make sure the all the binaries are on the mirror before we
> land
> > this but the this looks good to me otherwise.
> 
> Hi Sam, I've been working this morning to update the python_modules
subdirectory
> to work with the new build system, and I'm running into an odd problem.  One
of
> the modules, python-host (formerly python_host) shouldn't be installing
> anywhere, and so during the PAYLOAD inspection it's failing as it finds all
the
> non-normalized posix paths.  Is it possible to disable the installation step
> somehow?  (The specific error is "Invalid Package file")

You can setup and empty InstallStep()  (one that just contains a return
statement).

[Matthew Turk, 2014-04-08 23:23:26]

I've updated to match the new build system, and I believe it's now correctly
working -- I am able to build with TOOLCHAIN=pnacl NACL_ARCH=pnacl and get a
working pexe file.  Thanks for the help, Sam and Kester!

[Sam Clegg, 2014-04-10 19:29:50]

Mirrors have been updated.

Trying this stuff locally now.

https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/j...
File ports/python_modules/jinja2/pkg_info (right):

https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/j...
ports/python_modules/jinja2/pkg_info:1: NAME=Jinja2
name should match directory name.. so lower case 'j'.

[Sam Clegg, 2014-04-10 19:31:48]

lgtm

https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pk...
File ports/python-static/pkg_info (right):

https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pk...
ports/python-static/pkg_info:7: DEPENDS=(readline zlib glibc-compat pyppapi
openssl)
Seems to be missing bzip2 dependency.

I'm working on a change that will have the builder un-install everything between
each package build so this kind of thing will always get caught (at least by
builtbot).

[Sam Clegg, 2014-04-10 20:39:48]

On 2014/04/10 19:29:50, Sam Clegg wrote:
> Mirrors have been updated.
> 
> Trying this stuff locally now.
> 
>
https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/j...
> File ports/python_modules/jinja2/pkg_info (right):
> 
>
https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/j...
> ports/python_modules/jinja2/pkg_info:1: NAME=Jinja2
> name should match directory name.. so lower case 'j'.

When you do this you will also need to add ARCHIVE_ROOT=Jinja2-2.7.2  

Also, I had issues building this port:

creating
/usr/local/google/home/sbc/nacl_sdk/pepper_canary/toolchain/linux_x86_newlib/x86_64-nacl/usr/lib/python2.7/site-packages/Jinja2-2.7.2-py2.7.egg
Extracting Jinja2-2.7.2-py2.7.egg to
/usr/local/google/home/sbc/nacl_sdk/pepper_canary/toolchain/linux_x86_newlib/x86_64-nacl/usr/lib/python2.7/site-packages
Adding Jinja2 2.7.2 to easy-install.pth file

Installed
/usr/local/google/home/sbc/nacl_sdk/pepper_canary/toolchain/linux_x86_newlib/x86_64-nacl/usr/lib/python2.7/site-packages/Jinja2-2.7.2-py2.7.egg
Processing dependencies for Jinja2==2.7.2
Searching for markupsafe
Reading http://pypi.python.org/simple/markupsafe/
Download error on http://pypi.python.org/simple/markupsafe/: unknown url type:
https -- Some packages may not be found!
Reading http://pypi.python.org/simple/markupsafe/
Download error on http://pypi.python.org/simple/markupsafe/: unknown url type:
https -- Some packages may not be found!
Couldn't find index page for 'markupsafe' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
Download error on http://pypi.python.org/simple/: unknown url type: https --
Some packages may not be found!
No local packages or download links found for markupsafe
error: Could not find suitable distribution for Requirement.parse('markupsafe')
naclports: Building 'jinja2' failed.

[Matthew Turk, 2014-04-15 12:32:30]

This adds markupsafe and fixes the other issues sbc noted.

https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pk...
File ports/python-static/pkg_info (right):

https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pk...
ports/python-static/pkg_info:7: DEPENDS=(readline zlib glibc-compat pyppapi
openssl)
On 2014/04/10 19:31:48, Sam Clegg wrote:
> Seems to be missing bzip2 dependency.
> 
> I'm working on a change that will have the builder un-install everything
between
> each package build so this kind of thing will always get caught (at least by
> builtbot).

Done.

https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/j...
File ports/python_modules/jinja2/pkg_info (right):

https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/j...
ports/python_modules/jinja2/pkg_info:1: NAME=Jinja2
On 2014/04/10 19:29:51, Sam Clegg wrote:
> name should match directory name.. so lower case 'j'.

Done.

[Sam Clegg, 2014-04-15 22:15:49]

lgtm.

(somehow this issue was closes, so I re-opened it).

[Matthew Turk, 2014-04-15 23:12:49]

Committed patchset #11 manually as r1230.