Turn on --fast on official builds.

Turn on --fast on official builds.

The --fast support for build_packages is stable now, and should significantly
improve the speed of our official builders. We should turn it on so that
builders can finish their builds in less than 8 hours.

BUG=chromium-os:6706
TEST=Run ./enter_chroot.sh --official_build and verify that --fast is on by
     default now

Change-Id: I6ad126b9b6ce16ffc9887a7af22c2e3f85afbf42

Committed: http://chrome-svn/viewvc/chromeos?view=rev&revision=ff07201

[davidjames, 2010-11-30 19:22:43]

PTAL

[Nick Sanders, 2010-11-30 21:21:28]

lgtm

[anush, 2010-11-30 23:05:51]

Just for the record I was/am against turning this on by default for
official builds (for multiple reasons I have already gone over
multiple times and am not going to list it here). Speed is important,
accuracy more so. We have no data on why our parallel_emerge is better
than emerge --jobs, and data on how it outperforms the latter while
still preserving the locks. And what are the actual problems with
--jobs? Also are we sure we dont need an env-update after the package
is emerged?

 And just to be clear we should not be hostage to parallel_emerge not
working with the next portage update (since we dont seem interested in
fixing the problem but doing quick hacks to parallel_emerge), so
unless that is worked on we will revert to regular emerge or emerge
--jobs with the next portage update.

Thanks

On Tue, Nov 30, 2010 at 11:21 AM,  <nsanders@chromium.org> wrote:
> lgtm
>
> http://codereview.chromium.org/3418001/
>

[davidjames, 2010-12-01 00:17:16]

On 2010/11/30 23:05:51, anush wrote:
> Just for the record I was/am against turning this on by default for
> official builds (for multiple reasons I have already gone over
> multiple times and am not going to list it here). Speed is important,
> accuracy more so. We have no data on why our parallel_emerge is better
> than emerge --jobs, and data on how it outperforms the latter while
> still preserving the locks. And what are the actual problems with
> --jobs? Also are we sure we dont need an env-update after the package
> is emerged?
> 
>  And just to be clear we should not be hostage to parallel_emerge not
> working with the next portage update (since we dont seem interested in
> fixing the problem but doing quick hacks to parallel_emerge), so
> unless that is worked on we will revert to regular emerge or emerge
> --jobs with the next portage update.

Hi Anush,

I'm not sure I understand what you're getting at here. When we discussed this in
person, you said it would be fine for me to commit this as long as we switched
on the 'conservative mode' for official builds where all locks are preserved and
we are simply using the same logic as the original emerge program but with a
better scheduler that allows for parallel installs. That's exactly what we did
here. Perhaps you're thinking that this is a different change?

As for the problem with Portage's scheduler, the issue is that Portage is overly
aggressive -- it treats specific types of dependencies as 'optional' and
discards them if it speeds up the build. Unfortunately, discarding dependencies
often leads to incorrect builds so we end up with flakiness.

The flakiness with Portage --jobs wasn't a rare problem -- it hit us about half
of the time. The only way to avoid the flakiness was to run our jobs with
multiple retries and to retry without jobs when the flakiness was finished.

> We have no data on why our parallel_emerge is better
> than emerge --jobs, and data on how it outperforms the latter while
> still preserving the locks

We do in fact have data on this. make_chroot already uses the 'conservative
mode' for parallel_emerge and performs 2x as fast with parallel_emerge vs with
regular emerge. We benchmarked this change on the buildbots themselves and
noticed the improvements without any loss in reliability or accuracy. It's been
running successfully for several months now without problems.

As for upstreaming changes to Portage itself, I agree that is a nice goal. IIRC
we already have a portage developer on our team who is working on upstreaming
our modifications to Portage. I don't think we ever said that we weren't
interested in doing this.

Cheers,

David

[anush, 2010-12-01 17:28:54]

On Tue, Nov 30, 2010 at 2:17 PM,  <davidjames@chromium.org> wrote:
> On 2010/11/30 23:05:51, anush wrote:
>>
>> Just for the record I was/am against turning this on by default for
>> official builds (for multiple reasons I have already gone over
>> multiple times and am not going to list it here). Speed is important,
>> accuracy more so. We have no data on why our parallel_emerge is better
>> than emerge --jobs, and data on how it outperforms the latter while
>> still preserving the locks. And what are the actual problems with
>> --jobs? Also are we sure we dont need an env-update after the package
>> is emerged?
>
>>  And just to be clear we should not be hostage to parallel_emerge not
>> working with the next portage update (since we dont seem interested in
>> fixing the problem but doing quick hacks to parallel_emerge), so
>> unless that is worked on we will revert to regular emerge or emerge
>> --jobs with the next portage update.
>
> Hi Anush,
>
> I'm not sure I understand what you're getting at here. When we discussed
> this in
> person, you said it would be fine for me to commit this as long as we
> switched
> on the 'conservative mode' for official builds where all locks are preserved
> and
> we are simply using the same logic as the original emerge program but with a
> better scheduler that allows for parallel installs. That's exactly what we
> did
> here. Perhaps you're thinking that this is a different change?

Nope it is the same change. We wanted to have env-updated just like in
the regular emerge.

>
> As for the problem with Portage's scheduler, the issue is that Portage is
> overly
> aggressive -- it treats specific types of dependencies as 'optional' and
> discards them if it speeds up the build. Unfortunately, discarding
> dependencies
> often leads to incorrect builds so we end up with flakiness.

So if that is the root cause of requiring parallel_emerge we should
fix the portage scheduler. We cannot maintain our own fork and try to
re-implement everything portage/emerge already does. Also please see
to Diego's email on where parallel_emerge fails with dependencies.

>
> The flakiness with Portage --jobs wasn't a rare problem -- it hit us about
> half
> of the time. The only way to avoid the flakiness was to run our jobs with
> multiple retries and to retry without jobs when the flakiness was finished.
>

The other solution I have been advocating for is, we fix the scheduler
the right way. If parallel_emerge scheduler is better then lets patch
portage to use our scheduler, instead of trying to re-implement all of
what portage does again inside parallel_emerge.

>> We have no data on why our parallel_emerge is better
>> than emerge --jobs, and data on how it outperforms the latter while
>> still preserving the locks
>
> We do in fact have data on this. make_chroot already uses the 'conservative
> mode' for parallel_emerge and performs 2x as fast with parallel_emerge vs
> with
> regular emerge. We benchmarked this change on the buildbots themselves and
> noticed the improvements without any loss in reliability or accuracy. It's
> been
> running successfully for several months now without problems.

This is comparing against regular emerge without jobs and
parallel_emerge. How does parallel_emerge perform against "emerge
--jobs" (if we fix deps issues) is the right comparison.

>
> As for upstreaming changes to Portage itself, I agree that is a nice goal.
> IIRC
> we already have a portage developer on our team who is working on
> upstreaming
> our modifications to Portage. I don't think we ever said that we weren't
> interested in doing this.

 If we have changes to portage folks can help up upstream it. It is
unlikely anything living outside in parallel_emerge will automatically
find its way up to portage.

So I would say we disable fast on the official build now, until we
have worked through atleast the emerge scheduler issues and
parallel_emerge deps calculation and env updates.  Please include me
in the review.

Thanks
Anush

>
> Cheers,
>
> David
>
> http://codereview.chromium.org/3418001/
>

[davidjames, 2010-12-01 23:51:16]

On 2010/12/01 17:28:54, anush wrote:
> On Tue, Nov 30, 2010 at 2:17 PM,  <mailto:davidjames@chromium.org> wrote:
> > On 2010/11/30 23:05:51, anush wrote:
> >>
> >> Just for the record I was/am against turning this on by default for
> >> official builds (for multiple reasons I have already gone over
> >> multiple times and am not going to list it here). Speed is important,
> >> accuracy more so. We have no data on why our parallel_emerge is better
> >> than emerge --jobs, and data on how it outperforms the latter while
> >> still preserving the locks. And what are the actual problems with
> >> --jobs? Also are we sure we dont need an env-update after the package
> >> is emerged?
> >
> >> &nbsp;And just to be clear we should not be hostage to parallel_emerge not
> >> working with the next portage update (since we dont seem interested in
> >> fixing the problem but doing quick hacks to parallel_emerge), so
> >> unless that is worked on we will revert to regular emerge or emerge
> >> --jobs with the next portage update.
> >
> > Hi Anush,
> >
> > I'm not sure I understand what you're getting at here. When we discussed
> > this in
> > person, you said it would be fine for me to commit this as long as we
> > switched
> > on the 'conservative mode' for official builds where all locks are preserved
> > and
> > we are simply using the same logic as the original emerge program but with a
> > better scheduler that allows for parallel installs. That's exactly what we
> > did
> > here. Perhaps you're thinking that this is a different change?
> 
> Nope it is the same change. We wanted to have env-updated just like in
> the regular emerge.

I think this is where we are misunderstanding each other.

If you look at my change, env-update is enabled when OFFICIAL is enabled. So the
change is exactly as you requested, I thought.

> This is comparing against regular emerge without jobs and
> parallel_emerge. How does parallel_emerge perform against "emerge
> --jobs" (if we fix deps issues) is the right comparison.

Comparing against emerge --jobs isn't fair either because it fails to build
pretty much every time. You can accomplish a complete build by retrying multiple
times, but it's sufficiently flaky that the build time is really increased quite
a bit by the failures.

Once we get emerge working with --jobs (and finishing a full from source build
reliably), then we can do a fair comparison.

> So I would say we disable fast on the official build now, until we
> have worked through atleast the emerge scheduler issues and
> parallel_emerge deps calculation and env updates.  Please include me
> in the review.

1) env updates is already enabled as discussed above
2) What is the dep calculation issue? In what situation does parallel_emerge
install the wrong packages?

The thread on the dev@ list is discussing a situation with an unresolved
dependency conflict -- emerge itself is outputting the following warning:

!!! One or more updates have been skipped due to a dependency conflict:

x11-drivers/opengles:0 for /build/tegra2_seaboard/

  ('ebuild', '/build/tegra2_seaboard/', 'x11-drivers/opengles-0.0.3-r1',
'merge') conflicts with
    <x11-drivers/opengles-0.0.2 required by ('installed',
'/build/tegra2_seaboard/', 'chromeos-base/board-devices-0.0.1-r3', 'nomerge')

Despite the above warning, both emerge and parallel_emerge seem to be handling
updates to use flags correctly on my machine. So I am curious what the issue is
here. If we have an issue we should save the logs for it so that we can analyze
it and fix it.

Here are the benchmarks for the buildbots before and after the switch to --fast:
  Before the change: build_packages + build_image = 4h16m45s + 23m49s = 4h40m34s
  After the change: build_packages + build_image = 1h38m45s + 7m44s = 1h46m14s

That's a 3x speedup. This is very helpful for developers who are looking for
feedback on whether a full build of their changes actually runs all the way
through.

Thanks,

David

[Nick Sanders, 2010-12-02 02:26:19]

Couple points:

* We're leaving all the locks in place, and deps are already calculated by
portage code. If parallel_emerge doesn't build correctly, emerge isn't going
to either, as in the example above.

* Updating portage without integrating with parallel emerge already isn't an
option, core SW development isn't possible at this point without parallel
emerge.

Do you want to meet up on chat of VC to discuss?

On Wed, Dec 1, 2010 at 3:51 PM, <davidjames@chromium.org> wrote:

> On 2010/12/01 17:28:54, anush wrote:
>
>> On Tue, Nov 30, 2010 at 2:17 PM,  <mailto:davidjames@chromium.org> wrote:
>> > On 2010/11/30 23:05:51, anush wrote:
>> >>
>> >> Just for the record I was/am against turning this on by default for
>> >> official builds (for multiple reasons I have already gone over
>> >> multiple times and am not going to list it here). Speed is important,
>> >> accuracy more so. We have no data on why our parallel_emerge is better
>> >> than emerge --jobs, and data on how it outperforms the latter while
>> >> still preserving the locks. And what are the actual problems with
>> >> --jobs? Also are we sure we dont need an env-update after the package
>> >> is emerged?
>> >
>> >> &nbsp;And just to be clear we should not be hostage to parallel_emerge
>> not
>>
>> >> working with the next portage update (since we dont seem interested in
>> >> fixing the problem but doing quick hacks to parallel_emerge), so
>> >> unless that is worked on we will revert to regular emerge or emerge
>> >> --jobs with the next portage update.
>> >
>> > Hi Anush,
>> >
>> > I'm not sure I understand what you're getting at here. When we discussed
>> > this in
>> > person, you said it would be fine for me to commit this as long as we
>> > switched
>> > on the 'conservative mode' for official builds where all locks are
>> preserved
>> > and
>> > we are simply using the same logic as the original emerge program but
>> with a
>> > better scheduler that allows for parallel installs. That's exactly what
>> we
>> > did
>> > here. Perhaps you're thinking that this is a different change?
>>
>
>  Nope it is the same change. We wanted to have env-updated just like in
>> the regular emerge.
>>
>
> I think this is where we are misunderstanding each other.
>
> If you look at my change, env-update is enabled when OFFICIAL is enabled.
> So the
> change is exactly as you requested, I thought.
>
>
>  This is comparing against regular emerge without jobs and
>> parallel_emerge. How does parallel_emerge perform against "emerge
>> --jobs" (if we fix deps issues) is the right comparison.
>>
>
> Comparing against emerge --jobs isn't fair either because it fails to build
> pretty much every time. You can accomplish a complete build by retrying
> multiple
> times, but it's sufficiently flaky that the build time is really increased
> quite
> a bit by the failures.
>
> Once we get emerge working with --jobs (and finishing a full from source
> build
> reliably), then we can do a fair comparison.
>
>
>  So I would say we disable fast on the official build now, until we
>> have worked through atleast the emerge scheduler issues and
>> parallel_emerge deps calculation and env updates.  Please include me
>> in the review.
>>
>
> 1) env updates is already enabled as discussed above
> 2) What is the dep calculation issue? In what situation does
> parallel_emerge
> install the wrong packages?
>
> The thread on the dev@ list is discussing a situation with an unresolved
> dependency conflict -- emerge itself is outputting the following warning:
>
> !!! One or more updates have been skipped due to a dependency conflict:
>
> x11-drivers/opengles:0 for /build/tegra2_seaboard/
>
>  ('ebuild', '/build/tegra2_seaboard/', 'x11-drivers/opengles-0.0.3-r1',
> 'merge') conflicts with
>    <x11-drivers/opengles-0.0.2 required by ('installed',
> '/build/tegra2_seaboard/', 'chromeos-base/board-devices-0.0.1-r3',
> 'nomerge')
>
> Despite the above warning, both emerge and parallel_emerge seem to be
> handling
> updates to use flags correctly on my machine. So I am curious what the
> issue is
> here. If we have an issue we should save the logs for it so that we can
> analyze
> it and fix it.
>
> Here are the benchmarks for the buildbots before and after the switch to
> --fast:
>  Before the change: build_packages + build_image = 4h16m45s + 23m49s =
> 4h40m34s
>  After the change: build_packages + build_image = 1h38m45s + 7m44s =
> 1h46m14s
>
> That's a 3x speedup. This is very helpful for developers who are looking
> for
> feedback on whether a full build of their changes actually runs all the way
> through.
>
> Thanks,
>
>
> David
>
> http://codereview.chromium.org/3418001/
>

[Mandeep Singh Baines, 2010-12-02 03:26:06]

On Wed, Dec 1, 2010 at 9:28 AM, Anush Elangovan(அனுஷ்)
<anush@chromium.org> wrote:
> On Tue, Nov 30, 2010 at 2:17 PM,  <davidjames@chromium.org> wrote:
>> On 2010/11/30 23:05:51, anush wrote:
>>>
>>> Just for the record I was/am against turning this on by default for
>>> official builds (for multiple reasons I have already gone over
>>> multiple times and am not going to list it here). Speed is important,
>>> accuracy more so. We have no data on why our parallel_emerge is better
>>> than emerge --jobs, and data on how it outperforms the latter while
>>> still preserving the locks. And what are the actual problems with
>>> --jobs? Also are we sure we dont need an env-update after the package
>>> is emerged?
>>
>>>  And just to be clear we should not be hostage to parallel_emerge not
>>> working with the next portage update (since we dont seem interested in
>>> fixing the problem but doing quick hacks to parallel_emerge), so
>>> unless that is worked on we will revert to regular emerge or emerge
>>> --jobs with the next portage update.
>>
>> Hi Anush,
>>
>> I'm not sure I understand what you're getting at here. When we discussed
>> this in
>> person, you said it would be fine for me to commit this as long as we
>> switched
>> on the 'conservative mode' for official builds where all locks are preserved
>> and
>> we are simply using the same logic as the original emerge program but with a
>> better scheduler that allows for parallel installs. That's exactly what we
>> did
>> here. Perhaps you're thinking that this is a different change?
>
> Nope it is the same change. We wanted to have env-updated just like in
> the regular emerge.
>

FWIW, I don't think env-update works when cross-compiling.

>>
>> As for the problem with Portage's scheduler, the issue is that Portage is
>> overly
>> aggressive -- it treats specific types of dependencies as 'optional' and
>> discards them if it speeds up the build. Unfortunately, discarding
>> dependencies
>> often leads to incorrect builds so we end up with flakiness.
>
> So if that is the root cause of requiring parallel_emerge we should
> fix the portage scheduler. We cannot maintain our own fork and try to
> re-implement everything portage/emerge already does. Also please see
> to Diego's email on where parallel_emerge fails with dependencies.
>
>>
>> The flakiness with Portage --jobs wasn't a rare problem -- it hit us about
>> half
>> of the time. The only way to avoid the flakiness was to run our jobs with
>> multiple retries and to retry without jobs when the flakiness was finished.
>>
>
> The other solution I have been advocating for is, we fix the scheduler
> the right way. If parallel_emerge scheduler is better then lets patch
> portage to use our scheduler, instead of trying to re-implement all of
> what portage does again inside parallel_emerge.
>
>>> We have no data on why our parallel_emerge is better
>>> than emerge --jobs, and data on how it outperforms the latter while
>>> still preserving the locks
>>
>> We do in fact have data on this. make_chroot already uses the 'conservative
>> mode' for parallel_emerge and performs 2x as fast with parallel_emerge vs
>> with
>> regular emerge. We benchmarked this change on the buildbots themselves and
>> noticed the improvements without any loss in reliability or accuracy. It's
>> been
>> running successfully for several months now without problems.
>
> This is comparing against regular emerge without jobs and
> parallel_emerge. How does parallel_emerge perform against "emerge
> --jobs" (if we fix deps issues) is the right comparison.
>
>>
>> As for upstreaming changes to Portage itself, I agree that is a nice goal.
>> IIRC
>> we already have a portage developer on our team who is working on
>> upstreaming
>> our modifications to Portage. I don't think we ever said that we weren't
>> interested in doing this.
>
>  If we have changes to portage folks can help up upstream it. It is
> unlikely anything living outside in parallel_emerge will automatically
> find its way up to portage.
>
> So I would say we disable fast on the official build now, until we
> have worked through atleast the emerge scheduler issues and
> parallel_emerge deps calculation and env updates.  Please include me
> in the review.
>
> Thanks
> Anush
>
>>
>> Cheers,
>>
>> David
>>
>> http://codereview.chromium.org/3418001/
>>
>

[anush, 2010-12-02 04:45:54]

On Wed, Dec 1, 2010 at 4:25 PM, Nick Sanders <nsanders@chromium.org> wrote:
> Couple points:
> * We're leaving all the locks in place, and deps are already calculated by
> portage code. If parallel_emerge doesn't build correctly, emerge isn't going
> to either, as in the example above.
> * Updating portage without integrating with parallel emerge already isn't an
> option, core SW development isn't possible at this point without parallel
> emerge.

So the question is why the apprehension to fix "emerge --jobs", while
we all agree that is the way forward and we use most of
emerge/portage?

> Do you want to meet up on chat of VC to discuss?

Anyways, I have scheduled a meeting tomorrow. Lets discuss it then.

>
> On Wed, Dec 1, 2010 at 3:51 PM, <davidjames@chromium.org> wrote:
>>
>> On 2010/12/01 17:28:54, anush wrote:
>>>
>>> On Tue, Nov 30, 2010 at 2:17 PM,  <mailto:davidjames@chromium.org> wrote:
>>> > On 2010/11/30 23:05:51, anush wrote:
>>> >>
>>> >> Just for the record I was/am against turning this on by default for
>>> >> official builds (for multiple reasons I have already gone over
>>> >> multiple times and am not going to list it here). Speed is important,
>>> >> accuracy more so. We have no data on why our parallel_emerge is better
>>> >> than emerge --jobs, and data on how it outperforms the latter while
>>> >> still preserving the locks. And what are the actual problems with
>>> >> --jobs? Also are we sure we dont need an env-update after the package
>>> >> is emerged?
>>> >
>>> >> &nbsp;And just to be clear we should not be hostage to parallel_emerge
>>> >> not
>>> >> working with the next portage update (since we dont seem interested in
>>> >> fixing the problem but doing quick hacks to parallel_emerge), so
>>> >> unless that is worked on we will revert to regular emerge or emerge
>>> >> --jobs with the next portage update.
>>> >
>>> > Hi Anush,
>>> >
>>> > I'm not sure I understand what you're getting at here. When we
>>> > discussed
>>> > this in
>>> > person, you said it would be fine for me to commit this as long as we
>>> > switched
>>> > on the 'conservative mode' for official builds where all locks are
>>> > preserved
>>> > and
>>> > we are simply using the same logic as the original emerge program but
>>> > with a
>>> > better scheduler that allows for parallel installs. That's exactly what
>>> > we
>>> > did
>>> > here. Perhaps you're thinking that this is a different change?
>>
>>> Nope it is the same change. We wanted to have env-updated just like in
>>> the regular emerge.
>>
>> I think this is where we are misunderstanding each other.
>>
>> If you look at my change, env-update is enabled when OFFICIAL is enabled.
>> So the
>> change is exactly as you requested, I thought.
>>
>>> This is comparing against regular emerge without jobs and
>>> parallel_emerge. How does parallel_emerge perform against "emerge
>>> --jobs" (if we fix deps issues) is the right comparison.
>>
>> Comparing against emerge --jobs isn't fair either because it fails to
>> build
>> pretty much every time. You can accomplish a complete build by retrying
>> multiple
>> times, but it's sufficiently flaky that the build time is really increased
>> quite
>> a bit by the failures.
>>
>> Once we get emerge working with --jobs (and finishing a full from source
>> build
>> reliably), then we can do a fair comparison.
>>
>>> So I would say we disable fast on the official build now, until we
>>> have worked through atleast the emerge scheduler issues and
>>> parallel_emerge deps calculation and env updates.  Please include me
>>> in the review.
>>
>> 1) env updates is already enabled as discussed above
>> 2) What is the dep calculation issue? In what situation does
>> parallel_emerge
>> install the wrong packages?
>>
>> The thread on the dev@ list is discussing a situation with an unresolved
>> dependency conflict -- emerge itself is outputting the following warning:
>>
>> !!! One or more updates have been skipped due to a dependency conflict:
>>
>> x11-drivers/opengles:0 for /build/tegra2_seaboard/
>>
>>  ('ebuild', '/build/tegra2_seaboard/', 'x11-drivers/opengles-0.0.3-r1',
>> 'merge') conflicts with
>>    <x11-drivers/opengles-0.0.2 required by ('installed',
>> '/build/tegra2_seaboard/', 'chromeos-base/board-devices-0.0.1-r3',
>> 'nomerge')
>>
>> Despite the above warning, both emerge and parallel_emerge seem to be
>> handling
>> updates to use flags correctly on my machine. So I am curious what the
>> issue is
>> here. If we have an issue we should save the logs for it so that we can
>> analyze
>> it and fix it.
>>
>> Here are the benchmarks for the buildbots before and after the switch to
>> --fast:
>>  Before the change: build_packages + build_image = 4h16m45s + 23m49s =
>> 4h40m34s
>>  After the change: build_packages + build_image = 1h38m45s + 7m44s =
>> 1h46m14s
>>
>> That's a 3x speedup. This is very helpful for developers who are looking
>> for
>> feedback on whether a full build of their changes actually runs all the
>> way
>> through.
>>
>> Thanks,
>>
>> David
>>
>> http://codereview.chromium.org/3418001/
>
>

[Nick Sanders, 2010-12-02 19:11:36]

On Wed, Dec 1, 2010 at 8:45 PM, Anush Elangovan(அனுஷ்)
<anush@chromium.org>wrote:

> On Wed, Dec 1, 2010 at 4:25 PM, Nick Sanders <nsanders@chromium.org>
> wrote:
> > Couple points:
> > * We're leaving all the locks in place, and deps are
> already calculated by
> > portage code. If parallel_emerge doesn't build correctly, emerge isn't
> going
> > to either, as in the example above.
> > * Updating portage without integrating with parallel emerge already isn't
> an
> > option, core SW development isn't possible at this point without parallel
> > emerge.
>
> So the question is why the apprehension to fix "emerge --jobs", while
> we all agree that is the way forward and we use most of
> emerge/portage?
>

I don't think there's any consensus here. I'd prefer to upstream parallel
emerge, as it's both faster
and more strict on dependencies. We'd benefit open source most by sharing
our existing, well tested work.


>
> > Do you want to meet up on chat of VC to discuss?
>
> Anyways, I have scheduled a meeting tomorrow. Lets discuss it then.
>
> >
> > On Wed, Dec 1, 2010 at 3:51 PM, <davidjames@chromium.org> wrote:
> >>
> >> On 2010/12/01 17:28:54, anush wrote:
> >>>
> >>> On Tue, Nov 30, 2010 at 2:17 PM,  <mailto:davidjames@chromium.org>
> wrote:
> >>> > On 2010/11/30 23:05:51, anush wrote:
> >>> >>
> >>> >> Just for the record I was/am against turning this on by default for
> >>> >> official builds (for multiple reasons I have already gone over
> >>> >> multiple times and am not going to list it here). Speed is
> important,
> >>> >> accuracy more so. We have no data on why our parallel_emerge is
> better
> >>> >> than emerge --jobs, and data on how it outperforms the latter while
> >>> >> still preserving the locks. And what are the actual problems with
> >>> >> --jobs? Also are we sure we dont need an env-update after the
> package
> >>> >> is emerged?
> >>> >
> >>> >> &nbsp;And just to be clear we should not be hostage to
> parallel_emerge
> >>> >> not
> >>> >> working with the next portage update (since we dont seem interested
> in
> >>> >> fixing the problem but doing quick hacks to parallel_emerge), so
> >>> >> unless that is worked on we will revert to regular emerge or emerge
> >>> >> --jobs with the next portage update.
> >>> >
> >>> > Hi Anush,
> >>> >
> >>> > I'm not sure I understand what you're getting at here. When we
> >>> > discussed
> >>> > this in
> >>> > person, you said it would be fine for me to commit this as long as we
> >>> > switched
> >>> > on the 'conservative mode' for official builds where all locks are
> >>> > preserved
> >>> > and
> >>> > we are simply using the same logic as the original emerge program but
> >>> > with a
> >>> > better scheduler that allows for parallel installs. That's exactly
> what
> >>> > we
> >>> > did
> >>> > here. Perhaps you're thinking that this is a different change?
> >>
> >>> Nope it is the same change. We wanted to have env-updated just like in
> >>> the regular emerge.
> >>
> >> I think this is where we are misunderstanding each other.
> >>
> >> If you look at my change, env-update is enabled when OFFICIAL is
> enabled.
> >> So the
> >> change is exactly as you requested, I thought.
> >>
> >>> This is comparing against regular emerge without jobs and
> >>> parallel_emerge. How does parallel_emerge perform against "emerge
> >>> --jobs" (if we fix deps issues) is the right comparison.
> >>
> >> Comparing against emerge --jobs isn't fair either because it fails to
> >> build
> >> pretty much every time. You can accomplish a complete build by retrying
> >> multiple
> >> times, but it's sufficiently flaky that the build time is really
> increased
> >> quite
> >> a bit by the failures.
> >>
> >> Once we get emerge working with --jobs (and finishing a full from source
> >> build
> >> reliably), then we can do a fair comparison.
> >>
> >>> So I would say we disable fast on the official build now, until we
> >>> have worked through atleast the emerge scheduler issues and
> >>> parallel_emerge deps calculation and env updates.  Please include me
> >>> in the review.
> >>
> >> 1) env updates is already enabled as discussed above
> >> 2) What is the dep calculation issue? In what situation does
> >> parallel_emerge
> >> install the wrong packages?
> >>
> >> The thread on the dev@ list is discussing a situation with an
> unresolved
> >> dependency conflict -- emerge itself is outputting the following
> warning:
> >>
> >> !!! One or more updates have been skipped due to a dependency conflict:
> >>
> >> x11-drivers/opengles:0 for /build/tegra2_seaboard/
> >>
> >>  ('ebuild', '/build/tegra2_seaboard/', 'x11-drivers/opengles-0.0.3-r1',
> >> 'merge') conflicts with
> >>    <x11-drivers/opengles-0.0.2 required by ('installed',
> >> '/build/tegra2_seaboard/', 'chromeos-base/board-devices-0.0.1-r3',
> >> 'nomerge')
> >>
> >> Despite the above warning, both emerge and parallel_emerge seem to be
> >> handling
> >> updates to use flags correctly on my machine. So I am curious what the
> >> issue is
> >> here. If we have an issue we should save the logs for it so that we can
> >> analyze
> >> it and fix it.
> >>
> >> Here are the benchmarks for the buildbots before and after the switch to
> >> --fast:
> >>  Before the change: build_packages + build_image = 4h16m45s + 23m49s =
> >> 4h40m34s
> >>  After the change: build_packages + build_image = 1h38m45s + 7m44s =
> >> 1h46m14s
> >>
> >> That's a 3x speedup. This is very helpful for developers who are looking
> >> for
> >> feedback on whether a full build of their changes actually runs all the
> >> way
> >> through.
> >>
> >> Thanks,
> >>
> >> David
> >>
> >> http://codereview.chromium.org/3418001/
> >
> >
>