[crosutils] Add script to help build incrementally

[crosutils] Add script to help build incrementally

cros_workon_make provides a make-style interface to incrementally building
    cros_workon-enabled packages, appropriately setting up the toolchain for
    the target board beforehand.
    
    To incrementally build your package:
      cros_workon_make --board <board> <package>
    
    To re-run your package's configure steps and compile:
      cros_workon_make --board <board> <package> --reconf
    
    To incrementally build your package and tests, and run the tests:
      cros_workon_make --board <board> <package> --tests
    
    To incrementally build and install your package:
      cros_workon_make --board <board> <package> --install
    
    To remove ALL files in your package repo not known by git:
      cros_workon_make --board <board> <package> --scrub
    
    With the exception of --scrub, these flags all compose


BUG=6843
TEST=build workon packages in place

Change-Id: Ie219074127549e1e29adad6d6a03142ab74e0172

Committed: http://chrome-svn/viewvc/chromeos?view=rev&revision=5c5a95e

[Chris Masone, 2010-11-22 18:15:48]

On 2010/11/19 21:01:19, Chris Masone wrote:

ping?

[Chris Masone, 2010-11-23 06:46:43]

On 2010/11/22 18:15:48, Chris Masone wrote:
> On 2010/11/19 21:01:19, Chris Masone wrote:
> 
> ping?

PTAL

[anush, 2010-11-23 16:04:59]

Thanks for the update according to our conversation. 

High level question : Should these options be targets to ./cros_workon so
developer workflow becomes

./cros_workon start foo     #repo sync if necessary
./cros_workon make foo    #builds the default "install" target
./cros_workon [clean|reconf|inplace|distclean] foo
./cros_workon stop foo

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make
File bin/cros_workon_make (right):

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode26
bin/cros_workon_make:26: "Blow away all in-tree files not managed by git."
Should we call this "distclean" to keep with make'ish syntax ?

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode55
bin/cros_workon_make:55: die "run './cros_workon --board ${BOARD_STR} start
${1}' first!" 1>&2
I was wondering if we should just call the command we recommend running (and
repo sync), but then it kind of makes this tool do more than its current intent.
But that leads to the question if this script should just be a "make" target in
./cros_workon. So we run ./cros_workon start followed by ./cros_workon
make/build. Thoughts?

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode81
bin/cros_workon_make:81: if [ "${FLAGS_install}" = "${FLAGS_TRUE}" ]; then
Can we make this the default i.e without an "install" flag/target ? This makes
it possible for developers to build the default "all" target (i.e clean,
compile, test, install) . Then we can cover the workon in place with a flag.

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode93
bin/cros_workon_make:93: "${EBUILDCMD}" "${pkgfile}" ${clean} "${to_do}"
If we make the default as "install" then this needs to be wrapped into some
other target like "inplace" ?

[Chris Masone, 2010-11-23 19:42:21]

I'm against merging this into cros_workon for a couple reasons...one is personal
preference, the other is efficiency.  A one-file change a re-link takes maybe a
second or two for a package like chromeos-login.  Each equery takes 500ms -
700ms, which is a lot of overhead.  I wrote this tool to do just the one equery
in the normal path, so that we'd avoid wasting time in the common case.  I don't
see that being possible if I merged this into cros_workon.

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make
File bin/cros_workon_make (right):

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode26
bin/cros_workon_make:26: "Blow away all in-tree files not managed by git."
On 2010/11/23 16:04:59, anush wrote:
> Should we call this "distclean" to keep with make'ish syntax ?

I'm torn.  --scrub is really destructive, moreso than a distclean might be.  I
was considering adding an "are you sure" to --scrub, even.

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode55
bin/cros_workon_make:55: die "run './cros_workon --board ${BOARD_STR} start
${1}' first!" 1>&2
On 2010/11/23 16:04:59, anush wrote:
> I was wondering if we should just call the command we recommend running (and
> repo sync), but then it kind of makes this tool do more than its current
intent.
> But that leads to the question if this script should just be a "make" target
in
> ./cros_workon. So we run ./cros_workon start followed by ./cros_workon
> make/build. Thoughts?

I always prefer the toolbox approach to the swiss army knife approach.  You
might not WANT to repo sync after workon-ing the package.  But if you have the
minilayout, you might NEED to sync after workon-ing...I'd rather just keep each
tool simple and let people manage their own checkouts.

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode81
bin/cros_workon_make:81: if [ "${FLAGS_install}" = "${FLAGS_TRUE}" ]; then
On 2010/11/23 16:04:59, anush wrote:
> Can we make this the default i.e without an "install" flag/target ? This makes
> it possible for developers to build the default "all" target (i.e clean,
> compile, test, install) . Then we can cover the workon in place with a flag. 

No, the whole point of this tool is to enable incremental building.  If you want
to install, just emerge!

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode93
bin/cros_workon_make:93: "${EBUILDCMD}" "${pkgfile}" ${clean} "${to_do}"
On 2010/11/23 16:04:59, anush wrote:
> If we make the default as "install" then this needs to be wrapped into some
> other target like "inplace" ? 

nah, I think this should stay the default.  I'm writing this tool to enable an
incremental build workflow.  No one should really be using this to install
software anyway.  Just emerge it.

[anush, 2010-11-23 20:24:00]

Ok. LGTM to make it an incremental only script.

On Tue, Nov 23, 2010 at 11:42 AM,  <cmasone@chromium.org> wrote:
> I'm against merging this into cros_workon for a couple reasons...one is
> personal
> preference, the other is efficiency.  A one-file change a re-link takes
> maybe a
> second or two for a package like chromeos-login.  Each equery takes 500ms -
> 700ms, which is a lot of overhead.  I wrote this tool to do just the one
> equery
> in the normal path, so that we'd avoid wasting time in the common case.  I
> don't
> see that being possible if I merged this into cros_workon.
>
>
> http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make
> File bin/cros_workon_make (right):
>
>
http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode26
> bin/cros_workon_make:26: "Blow away all in-tree files not managed by
> git."
> On 2010/11/23 16:04:59, anush wrote:
>>
>> Should we call this "distclean" to keep with make'ish syntax ?
>
> I'm torn.  --scrub is really destructive, moreso than a distclean might
> be.  I was considering adding an "are you sure" to --scrub, even.
>
>
http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode55
> bin/cros_workon_make:55: die "run './cros_workon --board ${BOARD_STR}
> start ${1}' first!" 1>&2
> On 2010/11/23 16:04:59, anush wrote:
>>
>> I was wondering if we should just call the command we recommend
>
> running (and
>>
>> repo sync), but then it kind of makes this tool do more than its
>
> current intent.
>>
>> But that leads to the question if this script should just be a "make"
>
> target in
>>
>> ./cros_workon. So we run ./cros_workon start followed by ./cros_workon
>> make/build. Thoughts?
>
> I always prefer the toolbox approach to the swiss army knife approach.
> You might not WANT to repo sync after workon-ing the package.  But if
> you have the minilayout, you might NEED to sync after workon-ing...I'd
> rather just keep each tool simple and let people manage their own
> checkouts.
>
>
http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode81
> bin/cros_workon_make:81: if [ "${FLAGS_install}" = "${FLAGS_TRUE}" ];
> then
> On 2010/11/23 16:04:59, anush wrote:
>>
>> Can we make this the default i.e without an "install" flag/target ?
>
> This makes
>>
>> it possible for developers to build the default "all" target (i.e
>
> clean,
>>
>> compile, test, install) . Then we can cover the workon in place with a
>
> flag.
>
> No, the whole point of this tool is to enable incremental building.  If
> you want to install, just emerge!
>
>
http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode93
> bin/cros_workon_make:93: "${EBUILDCMD}" "${pkgfile}" ${clean} "${to_do}"
> On 2010/11/23 16:04:59, anush wrote:
>>
>> If we make the default as "install" then this needs to be wrapped into
>
> some
>>
>> other target like "inplace" ?
>
> nah, I think this should stay the default.  I'm writing this tool to
> enable an incremental build workflow.  No one should really be using
> this to install software anyway.  Just emerge it.
>
> http://codereview.chromium.org/5244001/
>

[rochberg, 2010-11-29 21:04:05]

I've been using CROS_WORKON_INPLACE and ran into trouble because nothing cleaned
up the .compiled file; ebuild ran into that and my build silently succeeded.  
I'm not sure I see where this script fixes that.

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make
File bin/cros_workon_make (right):

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode26
bin/cros_workon_make:26: "Blow away all in-tree files not managed by git."
+1 to "are you sure"---files you forgot to git add are the case that worries me.

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode93
bin/cros_workon_make:93: "${EBUILDCMD}" "${pkgfile}" ${clean} "${to_do}"
Since my incremental workflow involves the device, I end up installing (and then
scp'ing) to the device.  But I think that's an issue for a later CL.

http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode93
bin/cros_workon_make:93: "${EBUILDCMD}" "${pkgfile}" ${clean} "${to_do}"
Since my incremental workflow involves the device, I end up installing (and then
scp'ing) to the device.  But I think that's an issue for a later CL.

[Chris Masone, 2010-11-29 21:22:59]

On Mon, Nov 29, 2010 at 1:04 PM, <rochberg@chromium.org> wrote:

> I've been using CROS_WORKON_INPLACE and ran into trouble because nothing
> cleaned
> up the .compiled file; ebuild ran into that and my build silently
> succeeded.
> I'm not sure I see where this script fixes that.
>
>
in line 90, where it rm's the .compiled file.


>
>
>
> http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make
> File bin/cros_workon_make (right):
>
>
>
http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode26
> bin/cros_workon_make:26: "Blow away all in-tree files not managed by
> git."
> +1 to "are you sure"---files you forgot to git add are the case that
> worries me.
>
>
>
>
http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode93
> bin/cros_workon_make:93: "${EBUILDCMD}" "${pkgfile}" ${clean} "${to_do}"
> Since my incremental workflow involves the device, I end up installing
> (and then scp'ing) to the device.  But I think that's an issue for a
> later CL.
>
>
I'm strongly against the swiss army approach, where one tool gets an option
to support each individual's workflow.  This tool has a command to
incrementally build and install your package, though, which seems like it'd
get you most of what you need.


>
>
http://codereview.chromium.org/5244001/diff/8001/bin/cros_workon_make#newcode93
> bin/cros_workon_make:93: "${EBUILDCMD}" "${pkgfile}" ${clean} "${to_do}"
> Since my incremental workflow involves the device, I end up installing
> (and then scp'ing) to the device.  But I think that's an issue for a
> later CL.
>
>
> http://codereview.chromium.org/5244001/
>

[rochberg, 2010-11-30 02:44:52]

On Mon, Nov 29, 2010 at 4:22 PM, Chris Masone <cmasone@chromium.org> wrote:

>
>
> On Mon, Nov 29, 2010 at 1:04 PM, <rochberg@chromium.org> wrote:
>
>> I've been using CROS_WORKON_INPLACE and ran into trouble because nothing
>> cleaned
>> up the .compiled file; ebuild ran into that and my build silently
>> succeeded.
>> I'm not sure I see where this script fixes that.
>>
>>
> in line 90, where it rm's the .compiled file.
>

Perhaps I should learn to read.   Sorry.

>
>>
> I'm strongly against the swiss army approach, where one tool gets an option
> to support each individual's workflow.  This tool has a command to
> incrementally build and install your package, though, which seems like it'd
> get you most of what you need.
>

I totally agree that supporting every weird individual workflow is bad.
But I think there should be a "standard" fast way to get your bits to your
machine.  Once we have that, we can talk about where it belongs.  I'll start
a separate discussion about that.

[Chris Masone, 2010-11-30 02:51:04]

On Mon, Nov 29, 2010 at 6:44 PM, David Rochberg <rochberg@chromium.org>wrote:

> On Mon, Nov 29, 2010 at 4:22 PM, Chris Masone <cmasone@chromium.org>wrote:
>
>>
>>
>> On Mon, Nov 29, 2010 at 1:04 PM, <rochberg@chromium.org> wrote:
>>
>>> I've been using CROS_WORKON_INPLACE and ran into trouble because nothing
>>> cleaned
>>> up the .compiled file; ebuild ran into that and my build silently
>>> succeeded.
>>> I'm not sure I see where this script fixes that.
>>>
>>>
>> in line 90, where it rm's the .compiled file.
>>
>
> Perhaps I should learn to read.   Sorry.
>
>>
>>>
>> I'm strongly against the swiss army approach, where one tool gets an
>> option to support each individual's workflow.  This tool has a command to
>> incrementally build and install your package, though, which seems like it'd
>> get you most of what you need.
>>
>
> I totally agree that supporting every weird individual workflow is bad.
> But I think there should be a "standard" fast way to get your bits to your
> machine.  Once we have that, we can talk about where it belongs.  I'll start
> a separate discussion about that.
>

>

Don't we already have two?  gmerge and gmergefs?

I also think that, if one is installing onto the device so much that the
time it takes is a significant impediment to getting work done, the cycles
need to be spent to enable better unit testing, so more work can be done on
the workstation -- not on doing something that's just incrementally faster
than gmerge.

[kliegs, 2010-11-30 14:45:49]

On Mon, Nov 29, 2010 at 9:50 PM, Chris Masone <cmasone@chromium.org> wrote:

>
>
> On Mon, Nov 29, 2010 at 6:44 PM, David Rochberg <rochberg@chromium.org>wrote:
>
>> On Mon, Nov 29, 2010 at 4:22 PM, Chris Masone <cmasone@chromium.org>wrote:
>>
>>>
>>>
>>> On Mon, Nov 29, 2010 at 1:04 PM, <rochberg@chromium.org> wrote:
>>>
>>>> I've been using CROS_WORKON_INPLACE and ran into trouble because nothing
>>>> cleaned
>>>> up the .compiled file; ebuild ran into that and my build silently
>>>> succeeded.
>>>> I'm not sure I see where this script fixes that.
>>>>
>>>>
>>> in line 90, where it rm's the .compiled file.
>>>
>>
>> Perhaps I should learn to read.   Sorry.
>>
>>>
>>>>
>>> I'm strongly against the swiss army approach, where one tool gets an
>>> option to support each individual's workflow.  This tool has a command to
>>> incrementally build and install your package, though, which seems like it'd
>>> get you most of what you need.
>>>
>>
>> I totally agree that supporting every weird individual workflow is bad.
>> But I think there should be a "standard" fast way to get your bits to your
>> machine.  Once we have that, we can talk about where it belongs.  I'll start
>> a separate discussion about that.
>>
>
>>
>
> Don't we already have two?  gmerge and gmergefs?
>
> I also think that, if one is installing onto the device so much that the
> time it takes is a significant impediment to getting work done, the cycles
> need to be spent to enable better unit testing, so more work can be done on
> the workstation -- not on doing something that's just incrementally faster
> than gmerge.
>


I agree this would work for many tools.  But all the work on kernel drivers,
boot sequences and other random hardware bits will likely require testing on
the actual board. And I'm sure some other areas I've forgotten or weren't
aware of.  VM's only go so far for some instances and while being able to
unit test everything from host is a great goal, we don't want to ignore
improvements to other areas because "they should just unit test better".

I understand the reluctance to have one tool crammed full of so many
features its unmaintainable.  But we need to watch out for the other extreme
of having so many tools that we make the workflow too complex.

[Chris Masone, 2010-11-30 16:22:19]

On Tue, Nov 30, 2010 at 6:45 AM, Jonathan Kliegman <kliegs@chromium.org>wrote:

>
>
> On Mon, Nov 29, 2010 at 9:50 PM, Chris Masone <cmasone@chromium.org>wrote:
>
>>
>>
>> On Mon, Nov 29, 2010 at 6:44 PM, David Rochberg <rochberg@chromium.org>wrote:
>>
>>> On Mon, Nov 29, 2010 at 4:22 PM, Chris Masone <cmasone@chromium.org>wrote:
>>>
>>>>
>>>>
>>>> On Mon, Nov 29, 2010 at 1:04 PM, <rochberg@chromium.org> wrote:
>>>>
>>>>> I've been using CROS_WORKON_INPLACE and ran into trouble because
>>>>> nothing cleaned
>>>>> up the .compiled file; ebuild ran into that and my build silently
>>>>> succeeded.
>>>>> I'm not sure I see where this script fixes that.
>>>>>
>>>>>
>>>> in line 90, where it rm's the .compiled file.
>>>>
>>>
>>> Perhaps I should learn to read.   Sorry.
>>>
>>>>
>>>>>
>>>> I'm strongly against the swiss army approach, where one tool gets an
>>>> option to support each individual's workflow.  This tool has a command to
>>>> incrementally build and install your package, though, which seems like it'd
>>>> get you most of what you need.
>>>>
>>>
>>> I totally agree that supporting every weird individual workflow is bad.
>>> But I think there should be a "standard" fast way to get your bits to your
>>> machine.  Once we have that, we can talk about where it belongs.  I'll start
>>> a separate discussion about that.
>>>
>>
>>>
>>
>> Don't we already have two?  gmerge and gmergefs?
>>
>> I also think that, if one is installing onto the device so much that the
>> time it takes is a significant impediment to getting work done, the cycles
>> need to be spent to enable better unit testing, so more work can be done on
>> the workstation -- not on doing something that's just incrementally faster
>> than gmerge.
>>
>
>
> I agree this would work for many tools.  But all the work on kernel
> drivers, boot sequences and other random hardware bits will likely require
> testing on the actual board.
>

Absolutely.  And the two tools we have for installing software (gmerge,
gmergefs) certainly already support boot script work and all normal
packages.  If they don't work for the kernel or drivers, that sounds like
the place to add fixes.


> And I'm sure some other areas I've forgotten or weren't aware of.  VM's
> only go so far for some instances
>

I don't think VMs are in the discussion here...you'd need to gmerge stuff
onto them too, so it's the same as real devices in terms of
edit/compile/test.


> and while being able to unit test everything from host is a great goal, we
> don't want to ignore improvements to other areas because "they should just
> unit test better".
>
> I understand the reluctance to have one tool crammed full of so many
> features its unmaintainable.  But we need to watch out for the other extreme
> of having so many tools that we make the workflow too complex.
>
>

Everyone's already familiar with gmerge.  If that doesn't work for some use
cases, then let's make it work.