Add recipe to build CIPD package with relocatable Conda environment.

Add recipe to build CIPD package with relocatable Conda environment.

It butchers Conda directory before packaging to remove all references to local
installation prefix, thus making it fully relocatable (it will be installed into
another prefix on bots). This process downgrades it from conda environment into
"regular" portable python environment.

R=iannucci@chromium.org,dtu@chromium.org
BUG=457792
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298181

[Vadim Sh., 2015-12-10 05:57:31]

PTAL

Will finish Linux\Win part in a separate CL.

Dave, can you give me some (small) example of how you going to use scipy and
mathplotlib so I can verify they work from relocatable conda environment.

https://codereview.chromium.org/1511383003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/conda/resources/butcher_conda.py (right):

https://codereview.chromium.org/1511383003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:1: # Copyright
2015 The Chromium Authors. All rights reserved.
This file is horrible :)

Basically, I install conda env in a dir, and then do full text search for name
of this dir. Then do something with the files I find (remove, if I can).

All conda binaries (as far as I understand) are built in location independent
way already. When conda installs a package it looks for
"/opt/anaconda1anaconda2/anaconda3" in config files and replaces it with real
location of the prefix.

This script undoes this :)

[dtu, 2015-12-10 07:40:11]

> Dave, can you give me some (small) example of how you going to use scipy and
mathplotlib so I can verify they work from relocatable conda environment.


# Saves a blue histogram to test.png. May take a minute or two
# for matplotlib to load the font cache on the first run.
import numpy
from matplotlib import pyplot
from scipy import stats

assert stats.anderson_ksamp(([0,0,1,1], [2,2,3,3]))[2] < 0.01

x = numpy.random.randn(10000)
n, bins, patches = pyplot.hist(x, 50)
pyplot.savefig('test.png')

[Vadim Sh., 2015-12-10 19:34:40]

On 2015/12/10 07:40:11, dtu wrote:
> > Dave, can you give me some (small) example of how you going to use scipy and
> mathplotlib so I can verify they work from relocatable conda environment.
> 
> 
> # Saves a blue histogram to test.png. May take a minute or two
> # for matplotlib to load the font cache on the first run.
> import numpy
> from matplotlib import pyplot
> from scipy import stats
> 
> assert stats.anderson_ksamp(([0,0,1,1], [2,2,3,3]))[2] < 0.01
> 
> x = numpy.random.randn(10000)
> n, bins, patches = pyplot.hist(x, 50)
> pyplot.savefig('test.png')

Works on Mac. Will test on Linux and Win a bit later.

[Vadim Sh., 2015-12-10 22:44:12]

It seems to work on Linux too, but not as smoothly. Font config needs full path
name, and installation prefix is hardcoded in bunch of binaries  :(
(libfontconfig.so and libcrypto.so). Hate that linux stubborness to hardcode
absolute paths in binaries :-/

Well, numpy, scipy works. Mathplotlib generated the PNG file. Let's proceed.
Will try Windows next.

[Vadim Sh., 2015-12-11 01:10:47]

Robbie, please take a look. 

Made it work on Windows too. 240 Mb :-/

[iannucci, 2015-12-14 20:44:05]

generally lgtm with some comments/questions

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
File scripts/slave/recipe_modules/conda/api.py (right):

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:43: install_mode='copy')
wipe afterwards if it's destroyed anyway?

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:53: conda_exe = self.path.join('bin',
'conda')
I would make this a @property

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:57: env={'PYTHONPATH': ''})
I think {"PYTHONPATH": None} will delete it entirely.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:71: # Construct URL to installer. See
https://repo.continuum.io/miniconda/.
break out into @property?

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:87: # We acknowledge the license
agreement.
what if it changes? Should we archive this to cipd/gcs?

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
File scripts/slave/recipe_modules/conda/resources/butcher_conda.py (right):

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:56: 'qt.conf',    
    # Qt manages to work without it
it may be missing resources, depending on the contents. IIRC, this file
specifies where to find crap like icons and fonts and stuff.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:64: return
f.read(1024).startswith('#!' + bin_dir)
wouldn't '#!/' be enough to determine naughtiness?

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:98: # We aren't
going to build Tk\Tcl extensions, it's not 80s.
http://45.media.tumblr.com/f01045dddb5a403dd140e0a6d5b4422a/tumblr_mxqq53hg5c...

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:172: fake_prefix =
'/opt/fake-python-prefix'
why not just FAKE_PYTHON_PREFIX, which we can then s//<real_path> at
unpack-time? I guess that requires modifying a cipd-installed file.. ick.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipes/...
File scripts/slave/recipes/infra/build_conda_cipd_pkg.py (right):

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipes/...
scripts/slave/recipes/infra/build_conda_cipd_pkg.py:39: ]
should we pull these from the wheelhouse? or pin them somehow at least?

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipes/...
scripts/slave/recipes/infra/build_conda_cipd_pkg.py:53: cipd_pkg_name =
'infra/conda_python/scientific/' + api.cipd.platform_suffix()
why not `infra/conda_python/Science!/platform`? :p

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipes/...
scripts/slave/recipes/infra/build_conda_cipd_pkg.py:79: conda.wipe()
maybe worth making this a context managed thing so that the wipe is automatic?

with api.conda.env(CONDA_VERSION, staging_dir) as conda:
  .. stuff

[Vadim Sh., 2016-01-09 00:59:45]

It's ugly as hell and I don't satisfied at all... I think I'll want to remove
all hacks. But it'll take some time, and I don't have cycles to do it in a
proper way now, but perf guys need it now, so gonna land as is (as long as it
works).

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
File scripts/slave/recipe_modules/conda/api.py (right):

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:43: install_mode='copy')
On 2015/12/14 20:44:05, iannucci wrote:
> wipe afterwards if it's destroyed anyway?

The environment still can be used to call scripts etc, it just no longer proper
conda environment. Changed some wording.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:53: conda_exe = self.path.join('bin',
'conda')
On 2015/12/14 20:44:05, iannucci wrote:
> I would make this a @property

Done.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:57: env={'PYTHONPATH': ''})
On 2015/12/14 20:44:04, iannucci wrote:
> I think {"PYTHONPATH": None} will delete it entirely.

Done.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:71: # Construct URL to installer. See
https://repo.continuum.io/miniconda/.
On 2015/12/14 20:44:05, iannucci wrote:
> break out into @property?

Not worth it. I can't imagine it to be used anywhere else.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/api.py:87: # We acknowledge the license
agreement.
On 2015/12/14 20:44:05, iannucci wrote:
> what if it changes? Should we archive this to cipd/gcs?

License text is inside the conda environment (part of final cipd package). Or
did you mean something else?

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
File scripts/slave/recipe_modules/conda/resources/butcher_conda.py (right):

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:56: 'qt.conf',    
    # Qt manages to work without it
On 2015/12/14 20:44:05, iannucci wrote:
> it may be missing resources, depending on the contents. IIRC, this file
> specifies where to find crap like icons and fonts and stuff.

Yeah.. QtAssistant started normally though, so whatever.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:64: return
f.read(1024).startswith('#!' + bin_dir)
On 2015/12/14 20:44:05, iannucci wrote:
> wouldn't '#!/' be enough to determine naughtiness?

hm? #!/bin/bash is acceptable script

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/conda/resources/butcher_conda.py:172: fake_prefix =
'/opt/fake-python-prefix'
On 2015/12/14 20:44:05, iannucci wrote:
> why not just FAKE_PYTHON_PREFIX, which we can then s//<real_path> at
> unpack-time? I guess that requires modifying a cipd-installed file.. ick.

That's what conda does itself actually. Except it hacks binary files too (I
cringed a bit when I saw how it is done).

I think I'll add post-install scripts support to CIPD and invoke conda's
routines directly. But it requires more cycles to implement...

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipes/...
File scripts/slave/recipes/infra/build_conda_cipd_pkg.py (right):

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipes/...
scripts/slave/recipes/infra/build_conda_cipd_pkg.py:39: ]
On 2015/12/14 20:44:05, iannucci wrote:
> should we pull these from the wheelhouse? or pin them somehow at least?

This recipe will be running manually, very infrequently (probably once..).
Pinning and mirroring is unnecessary busywork in this case, imho.

https://codereview.chromium.org/1511383003/diff/140001/scripts/slave/recipes/...
scripts/slave/recipes/infra/build_conda_cipd_pkg.py:79: conda.wipe()
On 2015/12/14 20:44:05, iannucci wrote:
> maybe worth making this a context managed thing so that the wipe is automatic?
> 
> with api.conda.env(CONDA_VERSION, staging_dir) as conda:
>   .. stuff

Done.

[Vadim Sh., 2016-01-09 01:00:03]

The CQ bit was checked by vadimsh@chromium.org

[Vadim Sh., 2016-01-09 01:00:03]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1511383003/#ps160001
(title: " ")

[commit-bot: I haz the power, 2016-01-09 01:00:11]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1511383003/160001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1511383003/160001

[commit-bot: I haz the power, 2016-01-09 01:03:02]

Description was changed from

==========
Add recipe to build CIPD package with relocatable Conda environment.

It butchers Conda directory before packaging to remove all references to local
installation prefix, thus making it fully relocatable (it will be installed into
another prefix on bots). This process downgrades it from conda environment into
"regular" portable python environment.

R=iannucci@chromium.org,dtu@chromium.org
BUG=457792
==========

to

==========
Add recipe to build CIPD package with relocatable Conda environment.

It butchers Conda directory before packaging to remove all references to local
installation prefix, thus making it fully relocatable (it will be installed into
another prefix on bots). This process downgrades it from conda environment into
"regular" portable python environment.

R=iannucci@chromium.org,dtu@chromium.org
BUG=457792

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298181
==========

[commit-bot: I haz the power, 2016-01-09 01:03:02]

Committed patchset #9 (id:160001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=298181