This CL updates the parallel job library in the au test harness to be more robust.

This CL updates the parallel job library in the au test harness to be more robust.

Specifically, we migrate to using multiprocessing.Process rather than threading.Thread
which allows us to terminate pesky threads in the event a job takes too long.  In 
addition it allows us to remove some redundancy that was needed for threading.Thread.

I've also refactored the "waiting" code into the ParallelJob class and made it possible
to test with ctest --cache.

Change-Id: I7b691b676defb9d2140b006b5731874c7a08562c

BUG=chromium-os:11168, chromium-os:13027
TEST=Ran ctest on x86-generic using new CL logic that lets me do so.

Committed: http://chrome-svn/viewvc/chromeos?view=rev&revision=70e4ebf

[sosa, 2011-04-07 23:03:50]

Finally getting this out :)

[dgarrett, 2011-04-08 03:56:42]

Generally looks good, but a couple of questions. The main one is that it seems
weird that python semaphores would be cross-process, but it looks like you are
using them that way.

If you are certain that that's right, then LGTM.

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
File au_test_harness/parallel_test_job.py (right):

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:20: """Small wrapper for Process that
stores output of a its target method."""
its -> it's

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:22: MAX_TIMEOUT = 1800
Units?

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:122: job_start_semaphore =
threading.Semaphore(number_of_simultaneous_jobs)
Is this semaphore really safe across processes instead of threads?

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:125: for job, args in map(_TwoTupleize,
jobs, jobs_args):
_TwoTupleize could be replaced in-line with "lambda x, y: (x, y)" but that's
just a different style, not sure if it's better.

[sosa, 2011-04-08 18:59:04]

Gonna push.  Thanks for the review!

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
File au_test_harness/parallel_test_job.py (right):

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:20: """Small wrapper for Process that
stores output of a its target method."""
it's == it is :)

On 2011/04/08 03:56:42, dgarrett wrote:
> its -> it's

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:22: MAX_TIMEOUT = 1800
On 2011/04/08 03:56:42, dgarrett wrote:
> Units?

Done.

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:122: job_start_semaphore =
threading.Semaphore(number_of_simultaneous_jobs)
I think you're right, switching to multiprocessing.Semaphore instead and tested
with -j 2 and saw all 4 tests still get run.

On 2011/04/08 03:56:42, dgarrett wrote:
> Is this semaphore really safe across processes instead of threads?

http://codereview.chromium.org/6815003/diff/5001/au_test_harness/parallel_tes...
au_test_harness/parallel_test_job.py:125: for job, args in map(_TwoTupleize,
jobs, jobs_args):
On 2011/04/08 03:56:42, dgarrett wrote:
> _TwoTupleize could be replaced in-line with "lambda x, y: (x, y)" but that's
> just a different style, not sure if it's better.

Done.