Update autotest tests to use make -j where possible.

Update autotest tests to use make -j where possible.

This cuts the time required to build the autotests from 7m30s to
about 4m10s on my z600.

TEST=Ran ./autotest --build=all
BUG=chromium-os:5309

[zbehan, 2010-08-12 23:26:52]

Would it be possible to have this either generally respect MAKEOPTS instead of
doing the change specifically for number of cpu's, or at least have a
commandline option doing the same thing for autotest_client?

Also, would you be willing to make a patch against the upstream autotest?

[davidjames, 2010-08-13 00:09:16]

Updated to use MAKEOPTS per zbehan's comments.

[ericli, 2010-08-13 00:13:37]

would you could spit this into two CLs?

Things under site_test could be reviewed and submit here.

Other stuff please generate a raw patch and send it upstream.
autotest@test.kernel.org and jadmanski@google.com.

I will not LGTM this CL as it is.



On Thu, Aug 12, 2010 at 5:09 PM, <davidjames@chromium.org> wrote:

> Reviewers: zbehan, ericli,
>
> Message:
> Updated to use MAKEOPTS per zbehan's comments.
>
> Description:
> Update autotest tests to use make -j where possible.
>
> This cuts the time required to build the autotests from about 8 minutes to
> about 4 minutes on my z600.
>
> I mainly updated compile steps to use make -j. I didn't update the install
> steps, mainly because they aren't CPU-bound.
>
> TEST=Ran ./autotest --build=all
> BUG=chromium-os:5309
>
> Please review this at http://codereview.chromium.org/3116009/show
>
> SVN Base: ssh://git@chromiumos-git/autotest.git
>
> Affected files:
>  M client/bin/kernel_config.py
>  M client/bin/unit_test.py
>  M client/common_lib/utils.py
>  M client/deps/dejagnu/dejagnu.py
>  M client/deps/glbench/glbench.py
>  M client/deps/hdparm/hdparm.py
>  M client/deps/ibusclient/ibusclient.py
>  M client/deps/libaio/libaio.py
>  M client/deps/libcap/libcap.py
>  M client/deps/libnet/libnet.py
>  M client/deps/mysql/mysql.py
>  M client/deps/pgpool/pgpool.py
>  M client/deps/pgsql/pgsql.py
>  M client/deps/sysstat/sysstat.py
>  M client/deps/systemtap/systemtap.py
>  M client/profilers/blktrace/blktrace.py
>  M client/profilers/lockmeter/lockmeter.py
>  M client/profilers/lttng/lttng.py
>  M client/profilers/oprofile/oprofile.py
>  M client/profilers/powertop/powertop.py
>  M client/profilers/readprofile/readprofile.py
>  M
>
client/site_tests/audiovideo_PlaybackRecordSemiAuto/audiovideo_PlaybackRecordSemiAuto.py
>  M client/site_tests/audiovideo_V4L2/audiovideo_V4L2.py
>  M client/site_tests/firmware_VbootCrypto/firmware_VbootCrypto.py
>  M client/site_tests/graphics_GLAPICheck/graphics_GLAPICheck.py
>  M client/site_tests/graphics_SanAngeles/graphics_SanAngeles.py
>  M client/site_tests/hardware_MemoryThroughput/hardware_MemoryThroughput.py
>  M client/site_tests/hardware_SAT/hardware_SAT.py
>  M client/site_tests/hardware_TPM/hardware_TPM.py
>  M client/site_tests/hardware_TPMFirmware/hardware_TPMFirmware.py
>  M client/site_tests/hardware_bma150/hardware_bma150.py
>  M client/site_tests/logging_UserCrash/logging_UserCrash.py
>  M client/site_tests/network_netperf2/network_netperf2.py
>  M client/site_tests/platform_MiniJailCmdLine/platform_MiniJailCmdLine.py
>  M
>
client/site_tests/platform_MiniJailPidNamespace/platform_MiniJailPidNamespace.py
>  M
>
client/site_tests/platform_MiniJailPtraceDisabled/platform_MiniJailPtraceDisabled.py
>  M
> client/site_tests/platform_MiniJailReadOnlyFS/platform_MiniJailReadOnlyFS.py
>  M
>
client/site_tests/platform_MiniJailRootCapabilities/platform_MiniJailRootCapabilities.py
>  M client/site_tests/platform_MiniJailUidGid/platform_MiniJailUidGid.py
>  M
>
client/site_tests/platform_MiniJailVfsNamespace/platform_MiniJailVfsNamespace.py
>  M client/site_tests/platform_StackProtector/platform_StackProtector.py
>  M client/tests/dbench/dbench.py
>  M client/tests/disktest/disktest.py
>  M client/tests/iperf/iperf.py
>  M client/tests/netperf2/netperf2.py
>  M client/tests/netpipe/netpipe.py
>
>
>


-- 
Eric Li
李咏竹
Google Kirkland

[davidjames, 2010-08-13 00:52:26]

Hi Eric,

I've adjusted the CL so it only modifies files that are new in our repository
and do not exist upstream. This includes some files outside of site_tests that
have not been upstreamed yet. There is one exception to this: utils.py. The
change needs to modify utils.py in order to be committed because this is where
the 'make' function I added lives.

Cheers,

David

[ericli, 2010-08-13 16:36:34]

I dont have any issue with this change. I personally like this change since it
just get things faster. But I really like to push the same upstream first
culture.


If you dont mind to wait couple of days, I could help you to push the change on
utils.py to upstream first.






On 2010/08/13 00:52:26, davidjames wrote:
> Hi Eric,
> 
> I've adjusted the CL so it only modifies files that are new in our repository
> and do not exist upstream. This includes some files outside of site_tests that
> have not been upstreamed yet. There is one exception to this: utils.py. The
> change needs to modify utils.py in order to be committed because this is where
> the 'make' function I added lives.
> 
> Cheers,
> 
> David

[davidjames, 2010-08-13 20:23:46]

On 2010/08/13 16:36:34, ericli wrote:
> I dont have any issue with this change. I personally like this change since it
> just get things faster. But I really like to push the same upstream first
> culture.
> 
> 
> If you dont mind to wait couple of days, I could help you to push the change
on
> utils.py to upstream first.

Sure that would be great if you could help me get the patch to utils.py
upstreamed. Patch to just utils.py is here:
  http://codereview.chromium.org/download/issue3116009_15001_16002.diff

I hope to get this patch in to our tree in this iteration (iteration 10). That
means we have 1 week to get it in and meet our goal.

Cheers,

David

[zbehan, 2010-08-13 22:34:38]

I would suggest pushing both the tests changes and the utils.py change, even
though in two separate patches. More speed is good.

Even moreso, maybe you should check out the upstream autotest and see if
upstream has some more tests to fix.

On Fri, Aug 13, 2010 at 1:23 PM, <davidjames@chromium.org> wrote:

> On 2010/08/13 16:36:34, ericli wrote:
>
>> I dont have any issue with this change. I personally like this change
>> since it
>> just get things faster. But I really like to push the same upstream first
>> culture.
>>
>
>
>  If you dont mind to wait couple of days, I could help you to push the
>> change
>>
> on
>
>> utils.py to upstream first.
>>
>
> Sure that would be great if you could help me get the patch to utils.py
> upstreamed. Patch to just utils.py is here:
>  http://codereview.chromium.org/download/issue3116009_15001_16002.diff
>
> I hope to get this patch in to our tree in this iteration (iteration 10).
> That
> means we have 1 week to get it in and meet our goal.
>
>
> Cheers,
>
> David
>
>
> http://codereview.chromium.org/3116009/show
>

[ericli, 2010-08-13 22:37:31]

Please dont do this, and respect upstream first spirit.

Otherwise, you need to commit to help me to merge from upstream back to our
master and feel all the pains. And you need to make a long term commitment
to do that.

The upstream process is in progress for now.

On Fri, Aug 13, 2010 at 3:34 PM, Zdenek Behan <zbehan@chromium.org> wrote:

> I would suggest pushing both the tests changes and the utils.py change,
> even though in two separate patches. More speed is good.
>
> Even moreso, maybe you should check out the upstream autotest and see if
> upstream has some more tests to fix.
>
>
> On Fri, Aug 13, 2010 at 1:23 PM, <davidjames@chromium.org> wrote:
>
>> On 2010/08/13 16:36:34, ericli wrote:
>>
>>> I dont have any issue with this change. I personally like this change
>>> since it
>>> just get things faster. But I really like to push the same upstream first
>>> culture.
>>>
>>
>>
>>  If you dont mind to wait couple of days, I could help you to push the
>>> change
>>>
>> on
>>
>>> utils.py to upstream first.
>>>
>>
>> Sure that would be great if you could help me get the patch to utils.py
>> upstreamed. Patch to just utils.py is here:
>>  http://codereview.chromium.org/download/issue3116009_15001_16002.diff
>>
>> I hope to get this patch in to our tree in this iteration (iteration 10).
>> That
>> means we have 1 week to get it in and meet our goal.
>>
>>
>> Cheers,
>>
>> David
>>
>>
>> http://codereview.chromium.org/3116009/show
>>
>
>


-- 
Eric Li
李咏竹
Google Kirkland

[zbehan, 2010-08-13 22:42:11]

Sorry, i should have said pushing both utils.py and test changes TO
UPSTREAM. I definitely meant it.

On Fri, Aug 13, 2010 at 3:37 PM, Eric Li(李咏竹) <ericli@chromium.org> wrote:

> Please dont do this, and respect upstream first spirit.
>
> Otherwise, you need to commit to help me to merge from upstream back to our
> master and feel all the pains. And you need to make a long term commitment
> to do that.
>
> The upstream process is in progress for now.
>
> On Fri, Aug 13, 2010 at 3:34 PM, Zdenek Behan <zbehan@chromium.org> wrote:
>
>> I would suggest pushing both the tests changes and the utils.py change,
>> even though in two separate patches. More speed is good.
>>
>> Even moreso, maybe you should check out the upstream autotest and see if
>> upstream has some more tests to fix.
>>
>>
>> On Fri, Aug 13, 2010 at 1:23 PM, <davidjames@chromium.org> wrote:
>>
>>> On 2010/08/13 16:36:34, ericli wrote:
>>>
>>>> I dont have any issue with this change. I personally like this change
>>>> since it
>>>> just get things faster. But I really like to push the same upstream
>>>> first
>>>> culture.
>>>>
>>>
>>>
>>>  If you dont mind to wait couple of days, I could help you to push the
>>>> change
>>>>
>>> on
>>>
>>>> utils.py to upstream first.
>>>>
>>>
>>> Sure that would be great if you could help me get the patch to utils.py
>>> upstreamed. Patch to just utils.py is here:
>>>  http://codereview.chromium.org/download/issue3116009_15001_16002.diff
>>>
>>> I hope to get this patch in to our tree in this iteration (iteration 10).
>>> That
>>> means we have 1 week to get it in and meet our goal.
>>>
>>>
>>> Cheers,
>>>
>>> David
>>>
>>>
>>> http://codereview.chromium.org/3116009/show
>>>
>>
>>
>
>
> --
> Eric Li
> 李咏竹
> Google Kirkland
>
>
>

[ericli, 2010-08-18 03:16:52]

Upstream patch is done.
http://autotest.kernel.org/changeset/4748/trunk/client

I will merge it back to our repo tomorrow during normal business hours.

[zbehan, 2010-08-18 03:27:57]

Most awesome. Thanks David for making it, thanks Eric for helping to
upstream it.

On Tue, Aug 17, 2010 at 8:16 PM, <ericli@chromium.org> wrote:

> Upstream patch is done.
> http://autotest.kernel.org/changeset/4748/trunk/client
>
> I will merge it back to our repo tomorrow during normal business hours.
>
>
>
> http://codereview.chromium.org/3116009/show
>

[ericli, 2010-08-18 17:20:23]

This is done, you might revert your changes to client/bin/utils.py in this CL,
do a git pull and submit the rest of it.


On 2010/08/18 03:27:57, zbehan wrote:
> Most awesome. Thanks David for making it, thanks Eric for helping to
> upstream it.
> 
> On Tue, Aug 17, 2010 at 8:16 PM, <mailto:ericli@chromium.org> wrote:
> 
> > Upstream patch is done.
> > http://autotest.kernel.org/changeset/4748/trunk/client
> >
> > I will merge it back to our repo tomorrow during normal business hours.
> >
> >
> >
> > http://codereview.chromium.org/3116009/show
> >
>