Update parallel_emerge to support --workon.

Update parallel_emerge to support --workon.

Packages specified as workon packages are always built from source.
Dependencies of workon packages are also built from source.

Dependencies are calculated with --selective=n so that workon packages will be included in the install list. Packages that are not being worked on and are being unnecessarily replaced because of --selective=n are filtered out later.

This patch also fixes a bug with world file updating -- previously the world file was never updated because the packages were getting removed by RemoveInstalledPackages. We now keep the appropriate world file updates, and instead update SanitizeTree to handle world file updates without trouble.

I also optimized the cycle cracking algorithm so that it only visits each node once. This seems to improve run time significantly on graphs that have cycles.

TEST=Ran ./parallel_emerge -uDNvpg --board=x86-generic chromeos --workon=chromeos-chrome &&
         ./parallel_emerge -uDNvpg --board=x86-generic chromeos --workon=libcros &&
         ./parallel_emerge -uDNvpg world --workon=libxml2 &&
         ./parallel_emerge -uDNv -p world hard-host-depends --workon='bcel'
BUG=none

[Nick Sanders, 2010-07-10 23:09:00]

I'm not sure that what this is doing is always the right thing. I wonder if we
should have --workon, and --workon-deep or something? I know if my one line
printf change to nano caused 100 packages to build from source I would be sad.

http://codereview.chromium.org/2959006/diff/17001/18001
File parallel_emerge (right):

http://codereview.chromium.org/2959006/diff/17001/18001#newcode55
parallel_emerge:55: print "Dependencies of workon packages are also built from
source."
Can you ad usage or comment about when --workon should be called, like just from
./build_packages?

http://codereview.chromium.org/2959006/diff/17001/18001#newcode113
parallel_emerge:113: workon_set.update(shlex.split("
".join(shlex.split(workon_str))))
Is this going to leave quotes, or have trouble?
--workon="sys-apps/foo-1.1 chromeos-base/bar-2.2"

http://codereview.chromium.org/2959006/diff/17001/18001#newcode275
parallel_emerge:275: updatedep[fullpkg].setdefault("atom", "%s/%s" % (pkgdir,
pkgname))
Can you mention what an "atom" is here so people less familiar with portage
won't have to dig thru the portage code.. =(

http://codereview.chromium.org/2959006/diff/17001/18001#newcode444
parallel_emerge:444: if this_pkg["action"] == "world":
what's this do?

http://codereview.chromium.org/2959006/diff/17001/18001#newcode452
parallel_emerge:452: print "  %s -> %s " % (cycle[i], cycle[i+1])
would it be worth printing the deptype here?

http://codereview.chromium.org/2959006/diff/17001/18001#newcode737
parallel_emerge:737: for it in range(2):
Can you add a comment about why this is here? It's for adding cycle-cracking
ordering for workon packges, but I only know that because you told me earlier =)

[Nick Sanders, 2010-07-10 23:31:55]

http://codereview.chromium.org/2959006/diff/17001/18001
File parallel_emerge (right):

http://codereview.chromium.org/2959006/diff/17001/18001#newcode576
parallel_emerge:576: cmdline = (EmergeCommand() + " --nodeps --selective
--noreplace " +
what does --selective do?

http://codereview.chromium.org/2959006/diff/17001/18001#newcode586
parallel_emerge:586: cmdline += "--usepkg=n --getbinpkg=n --selective=n "
Comment on why for each arg?

http://codereview.chromium.org/2959006/diff/17001/18001#newcode727
parallel_emerge:727: PACKAGE, EMERGE_ARGS, BOARD, JOBS, WORKON =
ParseArgs(sys.argv)
maybe worth throwing these in a opts.packages opts.workon .. class, these guys
are kinda multiplying.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode727
parallel_emerge:727: PACKAGE, EMERGE_ARGS, BOARD, JOBS, WORKON =
ParseArgs(sys.argv)
or call them FLAGS_package etc.

[davidjames, 2010-07-11 03:09:41]

Updated. I also fixed parallel_emerge to parse regular emerge arguments
correctly, so we're able to figure out what the user is asking emerge to do.

http://codereview.chromium.org/2959006/diff/17001/18001
File parallel_emerge (right):

http://codereview.chromium.org/2959006/diff/17001/18001#newcode55
parallel_emerge:55: print "Dependencies of workon packages are also built from
source."
On 2010/07/10 23:09:00, Nick Sanders wrote:
> Can you ad usage or comment about when --workon should be called, like just
from
> ./build_packages?

Done.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode113
parallel_emerge:113: workon_set.update(shlex.split("
".join(shlex.split(workon_str))))
On 2010/07/10 23:09:00, Nick Sanders wrote:
> Is this going to leave quotes, or have trouble?
> --workon="sys-apps/foo-1.1 chromeos-base/bar-2.2"
> 

That test case seems to work for me.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode275
parallel_emerge:275: updatedep[fullpkg].setdefault("atom", "%s/%s" % (pkgdir,
pkgname))
On 2010/07/10 23:09:00, Nick Sanders wrote:
> Can you mention what an "atom" is here so people less familiar with portage
> won't have to dig thru the portage code.. =(

Done.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode452
parallel_emerge:452: print "  %s -> %s " % (cycle[i], cycle[i+1])
On 2010/07/10 23:09:00, Nick Sanders wrote:
> would it be worth printing the deptype here?

Done.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode576
parallel_emerge:576: cmdline = (EmergeCommand() + " --nodeps --selective
--noreplace " +
On 2010/07/10 23:31:56, Nick Sanders wrote:
> what does --selective do?

Added a comment.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode586
parallel_emerge:586: cmdline += "--usepkg=n --getbinpkg=n --selective=n "
On 2010/07/10 23:31:56, Nick Sanders wrote:
> Comment on why for each arg?

Done.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode727
parallel_emerge:727: PACKAGE, EMERGE_ARGS, BOARD, JOBS, WORKON =
ParseArgs(sys.argv)
On 2010/07/10 23:31:56, Nick Sanders wrote:
> maybe worth throwing these in a opts.packages opts.workon .. class, these guys
> are kinda multiplying.

Done.

http://codereview.chromium.org/2959006/diff/17001/18001#newcode737
parallel_emerge:737: for it in range(2):
On 2010/07/10 23:09:00, Nick Sanders wrote:
> Can you add a comment about why this is here? It's for adding cycle-cracking
> ordering for workon packges, but I only know that because you told me earlier
=)
> 

Done.

[Nick Sanders, 2010-07-13 06:17:46]

lgtm, looks great!