Introduce a dynamic process pool for the local test driver

Introduce a dynamic process pool for the local test driver

The new process pool allows adding jobs after testing has been started. It will also allow to restructure building the job queue (in a follow up CL), so that testing can start instantly while the queue is being built.

Also attempts to clean up the keyboard-interrupt logic. Idea: Only catch keyboard interrupt once per process at the outermost level. Use proper "finally" clauses to clean up everywhere where a keyboard interrupt might occur. Never turn named exceptions into none-exceptions using anonymous "raise".

TEST=python -m unittest pool_unittest
R=jkummerow@chromium.org

Committed: https://code.google.com/p/v8/source/detail?r=21310

[Michael Achenbach, 2014-05-13 13:16:25]

PTAL

[Jakob Kummerow, 2014-05-14 09:07:37]

LGTM with comments.

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/e...
File tools/testrunner/local/execution.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/e...
tools/testrunner/local/execution.py:88: # while the queue is filled.
Have you measured whether that would make a difference? Otherwise let's drop the
plan and the TODO.

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/p...
File tools/testrunner/local/pool.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/p...
tools/testrunner/local/pool.py:52: BUFFER = 100
As discussed, maybe make this worker_count * 4 or something like that?

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/p...
tools/testrunner/local/pool.py:55: self.workers = workers
nit: without context, "workers" make me expect a list of worker objects. I'd
prefer "num_workers" or "worker_count" (or "jobs" which we use elsewhere).

[Michael Achenbach, 2014-05-14 12:05:57]

Done. The total approach is ~0.5 seconds slower. I don't know why yet. The time
could be won back by starting to test earlier (see below).

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/e...
File tools/testrunner/local/execution.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/e...
tools/testrunner/local/execution.py:88: # while the queue is filled.
It takes 0.5 seconds... I rather leave the todo and make a measurement in a
follow up CL if this can be improved. If not, I'll remove the todo.

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/p...
File tools/testrunner/local/pool.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/p...
tools/testrunner/local/pool.py:52: BUFFER = 100
On 2014/05/14 09:07:37, Jakob wrote:
> As discussed, maybe make this worker_count * 4 or something like that?

Done.

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/p...
tools/testrunner/local/pool.py:55: self.workers = workers
On 2014/05/14 09:07:37, Jakob wrote:
> nit: without context, "workers" make me expect a list of worker objects. I'd
> prefer "num_workers" or "worker_count" (or "jobs" which we use elsewhere).

Done.

[Michael Achenbach, 2014-05-14 13:31:15]

Committed patchset #5 manually as r21310 (presubmit successful).