Refactor au_test_harness into modules and refactor to use worker design.

Refactor au_test_harness into modules and refactor to use worker design flow.

This CL is a large refactoring that moves the test_harness from using sub-classed classes of au_test into a one where we have the same AU_Test for all test flows but different test workers that operate on different types of machines.  Specifically we move the VM / Real-image specific logic into real_au_worker and vm_au_worker.  There isn't anything functionally different in this change.

Also, we move the classes into their own modules and clean up use of cros_build_lib to be more stylistically correct.

Change-Id: I3e25141174c3d5ba22962bf94365815e69e5bedf

BUG=chromium-os:11172
TEST=Ran with simple and full w/ w/out delta update payloads and with unittest and without unittest keys.  Ran real test using full suite on test device using full update payloads.

Committed: http://chrome-svn/viewvc/chromeos?view=rev&revision=5566eb1

[sosa, 2011-03-02 23:28:47]

PTAL

I know this is big, but retveld couldn't figure out how to do the diff of au
test harness ... maybe I missed a git mv.  I can try to figure out how to make
this look better.  Please advise.

[dgarrett, 2011-03-03 02:17:21]

This looks like a really good change to me. Pretty much all of my comments are
nit's, not real problems.

LGTM w/nits.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test.py
File bin/au_test_harness/au_test.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test....
bin/au_test_harness/au_test.py:26: @classmethod
Why use the classmethod decorator only for this method?

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test....
bin/au_test_harness/au_test.py:80: # start our proxy at a different. We then
tell our update tools to
s/different/different one/

(Yeah, I know it's my comment originally)

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test....
bin/au_test_harness/au_test.py:88: # This update is expected to fail...
Either this comment or the docstring is wrong.

(And yeah, my error originally)

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_worke...
File bin/au_test_harness/au_worker.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_worke...
bin/au_test_harness/au_worker.py:121: @classmethod
Why classmethod here?

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_worke...
bin/au_test_harness/au_worker.py:196: except Exception, e:
This syntax is morphing. What you are doing works, but is only documented in the
form:

"except Exception as e:"

I'm not really clear on what the difference is, but I think the 'as' is now
preferred.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/cros_au_...
File bin/au_test_harness/cros_au_test_harness.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/cros_au_...
bin/au_test_harness/cros_au_test_harness.py:1: #!/usr/bin/python
According to the new guide this should be:

#!/usr/bin/env python

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/cros_tes...
File bin/au_test_harness/cros_test_proxy.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/cros_tes...
bin/au_test_harness/cros_test_proxy.py:16: 
Thanks for cleaning up all this whitespace noise.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/dev_serv...
File bin/au_test_harness/dev_server_wrapper.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/dev_serv...
bin/au_test_harness/dev_server_wrapper.py:42: @classmethod
Again, not a problem, I'm just curious.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
File bin/au_test_harness/parallel_test_job.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:65: def
RunParallelJobs(number_of_sumultaneous_jobs, jobs, jobs_args,
s/sumultaneous/simultaneous/

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:67: 
Extra blank line.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:82: job_start_semaphore =
threading.Semaphore(number_of_sumultaneous_jobs)
s/sumultaneous/simultaneous/

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:97: # We use a semaphore to ensure we
don't run more jobs that required.
s/that/than/

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/vm_au_wo...
File bin/au_test_harness/vm_au_worker.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/vm_au_wo...
bin/au_test_harness/vm_au_worker.py:40: def __init__(self, options):
I thought init was supposed to be the first method, but not certain if that's
the spec.

[sosa, 2011-03-03 03:11:41]

Thanks for the review!!!

Pushing after another quick round of testing.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test.py
File bin/au_test_harness/au_test.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test....
bin/au_test_harness/au_test.py:26: @classmethod
Done to set class options i.e. static method since you can't override __init__
for unittests or even instantiate the class yourself (the unittest test harness
does it).

On 2011/03/03 02:17:21, dgarrett wrote:
> Why use the classmethod decorator only for this method?

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test....
bin/au_test_harness/au_test.py:80: # start our proxy at a different. We then
tell our update tools to
On 2011/03/03 02:17:21, dgarrett wrote:
> s/different/different one/
> 
> (Yeah, I know it's my comment originally)

Done.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_test....
bin/au_test_harness/au_test.py:88: # This update is expected to fail...
heh .. comment is wrong :) Done.

On 2011/03/03 02:17:21, dgarrett wrote:
> Either this comment or the docstring is wrong.
> 
> (And yeah, my error originally)

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_worke...
File bin/au_test_harness/au_worker.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_worke...
bin/au_test_harness/au_worker.py:121: @classmethod
Sets a static variable.  I'm using class variables instead of globals much like
with au_test.

On 2011/03/03 02:17:21, dgarrett wrote:
> Why classmethod here?

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/au_worke...
bin/au_test_harness/au_worker.py:196: except Exception, e:
On 2011/03/03 02:17:21, dgarrett wrote:
> This syntax is morphing. What you are doing works, but is only documented in
the
> form:
> 
> "except Exception as e:"
> 
> I'm not really clear on what the difference is, but I think the 'as' is now
> preferred.

Done.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/cros_au_...
File bin/au_test_harness/cros_au_test_harness.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/cros_au_...
bin/au_test_harness/cros_au_test_harness.py:1: #!/usr/bin/python
This hasn't been agreed upon and differs from google style.  I'd like that to be
vetted before I change it.

On 2011/03/03 02:17:21, dgarrett wrote:
> According to the new guide this should be:
> 
> #!/usr/bin/env python

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/dev_serv...
File bin/au_test_harness/dev_server_wrapper.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/dev_serv...
bin/au_test_harness/dev_server_wrapper.py:42: @classmethod
Static method.  Doesn't use any specific variables but associated with a dev
server.

On 2011/03/03 02:17:21, dgarrett wrote:
> Again, not a problem, I'm just curious.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
File bin/au_test_harness/parallel_test_job.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:65: def
RunParallelJobs(number_of_sumultaneous_jobs, jobs, jobs_args,
On 2011/03/03 02:17:21, dgarrett wrote:
> s/sumultaneous/simultaneous/

Done.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:67: 
On 2011/03/03 02:17:21, dgarrett wrote:
> Extra blank line.

Done.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:82: job_start_semaphore =
threading.Semaphore(number_of_sumultaneous_jobs)
On 2011/03/03 02:17:21, dgarrett wrote:
> s/sumultaneous/simultaneous/

Done.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/parallel...
bin/au_test_harness/parallel_test_job.py:97: # We use a semaphore to ensure we
don't run more jobs that required.
On 2011/03/03 02:17:21, dgarrett wrote:
> s/that/than/

Done.

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/vm_au_wo...
File bin/au_test_harness/vm_au_worker.py (right):

http://codereview.chromium.org/6597122/diff/5008/bin/au_test_harness/vm_au_wo...
bin/au_test_harness/vm_au_worker.py:40: def __init__(self, options):
Wasn't sure.  I put the helper functions it used above it ... but ok with moving
up.  Done.

On 2011/03/03 02:17:21, dgarrett wrote:
> I thought init was supposed to be the first method, but not certain if that's
> the spec.