Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(230)

Issue 138913004: Build system for statically-linked Python. (Closed)

Created:
6 years, 11 months ago by Matthew Turk
Modified:
6 years, 8 months ago
CC:
native-client-reviews_googlegroups.com
Visibility:
Public.

Description

Build system for statically-linked Python. This adds a new subdirectory, python_modules/ , and a mechanism for creating a statically linked Python build. It is designed primarily for PNaCl, where a lack of shared library support and the cost of translation mean that a large Python stack requires a complex, multi-step build process. This process can be summarized as such: * Build a host python containing all necessary build-time dependencies for the stack * Build each individual module * Collect object files and construct unified libpython2.7.a and python.pexe * Add a dynload module for statically linked modules When not automated, this is prone to error and difficult to add or remove additional modules. The system included here, while not perfect, addresses this by enabling individual modules to be selected and built ala carte, with a reubilding of python_static necessary to create the unified python.pexe and libpython2.7.a. The libpython2.7.a output by this can then be used as a link target for a python_ppapi-like module. As examples, and as components of a scientific python stack, this changelist includes: * numpy (host and destination builds) * cython (host-only build) * setuptools (host-only build) * matplotlib (dest-only build) * sympy (dest-only build) * ipython (dest-only build) * pyzmq (dest-only build) * h5py (dest-only build; requires hdf5, under separate CL review) * jinja2 (dest-only build) * pyppapi (dest-only build) * six (dest-only build) * distribute (dest-only build) * nose (dest-only build) * pyparsing (dest-only build) * dateutil (dest-only build) * markupsafe (dest-only build) * yt (dest-only build) This CL includes a port for the pyppapi module, which was developed in support of this and which provides a Python API to communicate between running Native Client modules and Chrome. This includes serialization of NumPy arrays into typed arrays as well as other Python/JS translations. Python module ports act very similarly to standard naclports modules, with two exceptions. The installation of the .py files has been moved into the building phase and the collection of .o files occurs during installation. Additionally, they utilize modules.libs and modules.list files. The first of these files should append to the variable PY_MOD_LINK to add on additional libraries for building the module, and the second is a list of compiled modules (not object files) produced by the python module which should be included in the symbol table in libpython2.7.a. For instance, if the compiled C module "fastcode" exists in a Python module and it requires linking against cfast, these files would look like: modules.list: fastcode modules.libs: PY_MOD_LINK+=" -lcfast" At present, this has been tested on PNaCl. To construct a Python build with a number of modules, a command like this can be issues: NACL_ARCH=pnacl make numpy sympy matplotlib python_static The resultant installed libpython2.7.a will contain all of those modules. Note that any pure-python modules auto-installed by setuptools will be functional in the resultant python destination, but any C-compiled python modules that have been auto-installed will simply fail. R=sbc@chromium.org, hamaji@chromium.org, kestert@google.com BUG= Committed: https://code.google.com/p/naclports/source/detail?r=1230

Patch Set 1 #

Total comments: 7

Patch Set 2 : Responding to comments, some cleanups. #

Patch Set 3 : Setting minimum SDK version #

Total comments: 12

Patch Set 4 : Responding to comments #

Total comments: 6

Patch Set 5 : Adding msse, dependencies and fixing MODLIBs #

Patch Set 6 : Updating to new build system #

Patch Set 7 : Fix dependencies and update with new build system. #

Patch Set 8 : Updating once more for build system #

Patch Set 9 : Update to match the new build system #

Total comments: 4

Patch Set 10 : Adding markupsafe and updating jinja2 #

Patch Set 11 : Final update with merge against current naclports.py #

Unified diffs Side-by-side diffs Delta from patch set Stats (+7621 lines, -6 lines) Patch
M Makefile View 1 2 3 4 5 6 7 8 1 chunk +1 line, -1 line 0 comments Download
M build_tools/common.sh View 1 2 3 4 5 6 7 8 9 10 4 chunks +45 lines, -3 lines 0 comments Download
M build_tools/naclports.py View 1 2 3 4 5 6 7 8 9 10 2 chunks +11 lines, -2 lines 0 comments Download
A ports/python-static/Setup.local View 1 2 3 4 5 6 7 8 1 chunk +102 lines, -0 lines 0 comments Download
A ports/python-static/build.sh View 1 2 3 4 5 6 7 8 1 chunk +74 lines, -0 lines 0 comments Download
A ports/python-static/config.site View 1 2 3 4 5 6 7 8 1 chunk +10 lines, -0 lines 0 comments Download
A ports/python-static/dynload_ppapi.c View 1 2 3 4 5 6 7 8 1 chunk +21 lines, -0 lines 0 comments Download
A ports/python-static/nacl.patch View 1 2 3 4 5 6 7 8 1 chunk +627 lines, -0 lines 0 comments Download
A ports/python-static/pkg_info View 1 2 3 4 5 6 7 8 9 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/cython/build.sh View 1 2 3 4 5 6 7 8 1 chunk +29 lines, -0 lines 0 comments Download
A ports/python_modules/cython/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/distribute/build.sh View 1 2 3 4 5 6 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/distribute/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/h5py/build.sh View 1 2 3 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/h5py/modules.libs View 1 chunk +1 line, -0 lines 0 comments Download
A ports/python_modules/h5py/modules.list View 1 chunk +22 lines, -0 lines 0 comments Download
A ports/python_modules/h5py/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/ipython/build.sh View 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/ipython/nacl.patch View 1 1 chunk +22 lines, -0 lines 0 comments Download
A ports/python_modules/ipython/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/jinja2/build.sh View 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/jinja2/pkg_info View 1 2 3 4 5 6 7 8 9 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/markupsafe/build.sh View 1 2 3 4 5 6 7 8 9 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/markupsafe/pkg_info View 1 2 3 4 5 6 7 8 9 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/matplotlib/build.sh View 1 chunk +17 lines, -0 lines 0 comments Download
A ports/python_modules/matplotlib/modules.libs View 1 chunk +1 line, -0 lines 0 comments Download
A ports/python_modules/matplotlib/modules.list View 1 chunk +9 lines, -0 lines 0 comments Download
A ports/python_modules/matplotlib/nacl.patch View 1 2 3 4 5 6 1 chunk +167 lines, -0 lines 0 comments Download
A ports/python_modules/matplotlib/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/nose/build.sh View 1 2 3 4 5 6 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/nose/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/numpy/build.sh View 1 2 3 4 5 6 7 8 1 chunk +34 lines, -0 lines 0 comments Download
A ports/python_modules/numpy/modules.list View 1 chunk +11 lines, -0 lines 0 comments Download
A ports/python_modules/numpy/nacl.patch View 1 2 3 1 chunk +174 lines, -0 lines 0 comments Download
A ports/python_modules/numpy/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/pyparsing/build.sh View 1 2 3 4 5 6 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/pyparsing/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/pyppapi/build.sh View 1 chunk +17 lines, -0 lines 0 comments Download
A ports/python_modules/pyppapi/modules.list View 1 chunk +1 line, -0 lines 0 comments Download
A ports/python_modules/pyppapi/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/python-dateutil/build.sh View 1 2 3 4 5 6 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/python-dateutil/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/python-host/build.sh View 1 2 3 4 5 6 7 8 1 chunk +27 lines, -0 lines 0 comments Download
A ports/python_modules/python-host/nacl.patch View 1 2 3 4 5 6 7 8 1 chunk +627 lines, -0 lines 0 comments Download
A ports/python_modules/python-host/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +8 lines, -0 lines 0 comments Download
A ports/python_modules/pyzmq/build.sh View 1 chunk +17 lines, -0 lines 0 comments Download
A ports/python_modules/pyzmq/modules.libs View 1 chunk +1 line, -0 lines 0 comments Download
A ports/python_modules/pyzmq/modules.list View 1 chunk +12 lines, -0 lines 0 comments Download
A ports/python_modules/pyzmq/nacl.patch View 1 chunk +5159 lines, -0 lines 0 comments Download
A ports/python_modules/pyzmq/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/setuptools/build.sh View 1 2 3 4 5 6 7 8 1 chunk +29 lines, -0 lines 0 comments Download
A ports/python_modules/setuptools/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/six/build.sh View 1 2 3 4 5 6 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/six/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/sympy/build.sh View 1 chunk +16 lines, -0 lines 0 comments Download
A ports/python_modules/sympy/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download
A ports/python_modules/yt/build.sh View 1 chunk +17 lines, -0 lines 0 comments Download
A ports/python_modules/yt/modules.list View 1 chunk +35 lines, -0 lines 0 comments Download
A ports/python_modules/yt/pkg_info View 1 2 3 4 5 6 7 8 1 chunk +7 lines, -0 lines 0 comments Download

Messages

Total messages: 24 (0 generated)
Matthew Turk
This changelist includes a number of new additions, primarily in the way of adding the ...
6 years, 11 months ago (2014-01-20 15:22:20 UTC) #1
Sam Clegg
This change looks really promising. I've not had time to look at it in depth ...
6 years, 11 months ago (2014-01-22 01:46:49 UTC) #2
Matthew Turk
Thanks for the comments, Sam! I've updated with those, and removed a few unrelated changes. ...
6 years, 11 months ago (2014-01-22 21:14:31 UTC) #3
Sam Clegg
https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython/nacl.patch File ports/python_modules/ipython/nacl.patch (right): https://codereview.chromium.org/138913004/diff/1/ports/python_modules/ipython/nacl.patch#newcode35 ports/python_modules/ipython/nacl.patch:35: -devnull = open(os.devnull, 'a') On 2014/01/22 21:14:31, Matthew Turk ...
6 years, 11 months ago (2014-01-22 22:22:26 UTC) #4
hamaji
Please note that I don't think I'm a good reviewer for naclports changes because I'm ...
6 years, 11 months ago (2014-01-24 14:32:48 UTC) #5
Matthew Turk
Thanks for the feedback. I've gone through and addressed the comments, except for the pyzmq/nacl.patch, ...
6 years, 10 months ago (2014-01-28 19:39:19 UTC) #6
Kester Tong
Hey, This looks really cool, thanks for this contribution. I've been trying to build this ...
6 years, 10 months ago (2014-01-29 20:38:47 UTC) #7
Matthew Turk
Thanks for the kind words and comments, Kester! I've updated with those changes now. I'm ...
6 years, 10 months ago (2014-01-31 15:56:56 UTC) #8
Kester Tong
Thanks for making those changes. I'm not sure about renaming files in a patch file ...
6 years, 10 months ago (2014-02-05 18:08:03 UTC) #9
Matthew Turk
This patchset updates to match the new build system, and includes a modification to enable ...
6 years, 10 months ago (2014-02-08 02:57:07 UTC) #10
Matthew Turk
This patch addresses all previous comments, applies against the current svn HEAD, and avoids the ...
6 years, 9 months ago (2014-03-03 19:50:02 UTC) #11
Matthew Turk
On 2014/03/03 19:50:02, Matthew Turk wrote: > This patch addresses all previous comments, applies against ...
6 years, 9 months ago (2014-03-03 20:05:03 UTC) #12
Matthew Turk
Sorry for the false alarm earlier -- this updates everything to the new build system, ...
6 years, 9 months ago (2014-03-04 01:30:51 UTC) #13
Matthew Turk
On 2014/03/04 01:30:51, Matthew Turk wrote: > Sorry for the false alarm earlier -- this ...
6 years, 8 months ago (2014-04-07 17:38:01 UTC) #14
Sam Clegg
On 2014/04/07 17:38:01, Matthew Turk wrote: > On 2014/03/04 01:30:51, Matthew Turk wrote: > > ...
6 years, 8 months ago (2014-04-07 18:25:32 UTC) #15
Matthew Turk
On 2014/04/07 18:25:32, Sam Clegg wrote: > On 2014/04/07 17:38:01, Matthew Turk wrote: > > ...
6 years, 8 months ago (2014-04-08 13:19:26 UTC) #16
Sam Clegg
On 2014/04/08 13:19:26, Matthew Turk wrote: > On 2014/04/07 18:25:32, Sam Clegg wrote: > > ...
6 years, 8 months ago (2014-04-08 15:38:30 UTC) #17
Matthew Turk
I've updated to match the new build system, and I believe it's now correctly working ...
6 years, 8 months ago (2014-04-08 23:23:26 UTC) #18
Sam Clegg
Mirrors have been updated. Trying this stuff locally now. https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/jinja2/pkg_info File ports/python_modules/jinja2/pkg_info (right): https://codereview.chromium.org/138913004/diff/1960001/ports/python_modules/jinja2/pkg_info#newcode1 ports/python_modules/jinja2/pkg_info:1: ...
6 years, 8 months ago (2014-04-10 19:29:50 UTC) #19
Sam Clegg
lgtm https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pkg_info File ports/python-static/pkg_info (right): https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pkg_info#newcode7 ports/python-static/pkg_info:7: DEPENDS=(readline zlib glibc-compat pyppapi openssl) Seems to be ...
6 years, 8 months ago (2014-04-10 19:31:48 UTC) #20
Sam Clegg
On 2014/04/10 19:29:50, Sam Clegg wrote: > Mirrors have been updated. > > Trying this ...
6 years, 8 months ago (2014-04-10 20:39:48 UTC) #21
Matthew Turk
This adds markupsafe and fixes the other issues sbc noted. https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pkg_info File ports/python-static/pkg_info (right): https://codereview.chromium.org/138913004/diff/1960001/ports/python-static/pkg_info#newcode7 ...
6 years, 8 months ago (2014-04-15 12:32:30 UTC) #22
Sam Clegg
lgtm. (somehow this issue was closes, so I re-opened it).
6 years, 8 months ago (2014-04-15 22:15:49 UTC) #23
Matthew Turk
6 years, 8 months ago (2014-04-15 23:12:49 UTC) #24
Message was sent while issue was closed.
Committed patchset #11 manually as r1230.

Powered by Google App Engine
This is Rietveld 408576698