Integrate parallel_emerge with emerge, boosting performance.

Integrate parallel_emerge with emerge, boosting performance.

Instead of loosely wrapping emerge, parallel_emerge now integrates tightly with emerge. This boosts performance while allowing us to map dependencies more accurately.

With this change, build_image --fast can clock under 4 minutes, and build_packages --fast now clocks as low as 4:30. With the --rebuild option, build_packages takes 5:30, but it has the big benefit of producing actually-correct results.

Note that this change also depends on us updating the various build scripts to prefix
calls to parallel_emerge with sudo -E.

TEST=Ran several parallel_emerge test cases, build_packages --fast, and
     build_image --fast
BUG=none

[Nick Sanders, 2010-07-13 06:03:39]

lgtm w/ changes

Basically emerge code is kind of hard to follow, and calling into emerge
functions and using internal data structures is even more so..  So it would be
really good to add a ton of comments everywhere for anyone who hasn't been
digging around the emerge code.

Also, one functional change, it should always return shell failure on fail, I
think..

http://codereview.chromium.org/2891013/diff/6001/7001
File parallel_emerge (right):

http://codereview.chromium.org/2891013/diff/6001/7001#newcode50
parallel_emerge:50: 
Add some kind of comment regarding the blatant rummaging around in portage's
_private guts here, and why it's still the best solution available.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode52
parallel_emerge:52: from _emerge.AtomArg import AtomArg
Can you add a path where these files can be found, since they are not on the
normal python path

http://codereview.chromium.org/2891013/diff/6001/7001#newcode124
parallel_emerge:124: if "board" in myopts:
Can you add docs on what these are for, and what references them.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode173
parallel_emerge:173: emerge_opts = EMERGE_OPTS.copy()
Docs on why these args.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode182
parallel_emerge:182: settings, trees, mtimedb = load_emerge_config()
Docs on what these functions do and return, since they are external and mostly
undocumented.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode197
parallel_emerge:197: sys.exit(0)
exit 1, we have failed.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode205
parallel_emerge:205: deps[str(child.cpv)] = dict(action=str(child.operation),
A comment about what's a "child" struct

http://codereview.chromium.org/2891013/diff/6001/7001#newcode212
parallel_emerge:212: if isinstance(node, AtomArg):
Why AtomArg? comment..

http://codereview.chromium.org/2891013/diff/6001/7001#newcode236
parallel_emerge:236: if old_cpv_split[0:2] == pkg.cpv_split[0:2]:
comment cpv format and example string

http://codereview.chromium.org/2891013/diff/6001/7001#newcode682
parallel_emerge:682: if portage.secpass < 2:
Quick comment why we want root access to cache.

[davidjames, 2010-07-20 05:16:23]

This should shave a minute or two off of build_packages and build_image. Check
it out.

http://codereview.chromium.org/2891013/diff/6001/7001
File parallel_emerge (right):

http://codereview.chromium.org/2891013/diff/6001/7001#newcode50
parallel_emerge:50: 
On 2010/07/13 06:03:39, Nick Sanders wrote:
> Add some kind of comment regarding the blatant rummaging around in portage's
> _private guts here, and why it's still the best solution available.

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode52
parallel_emerge:52: from _emerge.AtomArg import AtomArg
On 2010/07/13 06:03:39, Nick Sanders wrote:
> Can you add a path where these files can be found, since they are not on the
> normal python path

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode124
parallel_emerge:124: if "board" in myopts:
On 2010/07/13 06:03:39, Nick Sanders wrote:
> Can you add docs on what these are for, and what references them.

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode173
parallel_emerge:173: emerge_opts = EMERGE_OPTS.copy()
On 2010/07/13 06:03:39, Nick Sanders wrote:
> Docs on why these args.

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode182
parallel_emerge:182: settings, trees, mtimedb = load_emerge_config()
On 2010/07/13 06:03:39, Nick Sanders wrote:
> Docs on what these functions do and return, since they are external and mostly
> undocumented.

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode197
parallel_emerge:197: sys.exit(0)
On 2010/07/13 06:03:39, Nick Sanders wrote:
> exit 1, we have failed.

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode205
parallel_emerge:205: deps[str(child.cpv)] = dict(action=str(child.operation),
On 2010/07/13 06:03:39, Nick Sanders wrote:
> A comment about what's a "child" struct

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode212
parallel_emerge:212: if isinstance(node, AtomArg):
On 2010/07/13 06:03:39, Nick Sanders wrote:
> Why AtomArg? comment.. 

Removed.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode236
parallel_emerge:236: if old_cpv_split[0:2] == pkg.cpv_split[0:2]:
On 2010/07/13 06:03:39, Nick Sanders wrote:
> comment cpv format and example string

Done.

http://codereview.chromium.org/2891013/diff/6001/7001#newcode682
parallel_emerge:682: if portage.secpass < 2:
On 2010/07/13 06:03:39, Nick Sanders wrote:
> Quick comment why we want root access to cache. 

Done.

[Nick Sanders, 2010-07-20 05:43:40]

lgtm w/ comment below, plus sudo change

http://codereview.chromium.org/2891013/diff/11002/23001
File parallel_emerge (right):

http://codereview.chromium.org/2891013/diff/11002/23001#newcode1029
parallel_emerge:1029: print "Emerging", target
Can you mention that get is blocking