Parallelize run_corpus_tests.py.

Parallelize run_corpus_tests.py.

- Use the number of cores as the default -j value
- Fall back to old code for -j 1

R=nparker@chromium.org

Committed: https://pdfium.googlesource.com/pdfium/+/fd751f28cecce61ab36038799043639d570e0b26

[Lei Zhang, 2015-10-09 17:53:11]

thestig@chromium.org changed reviewers:
+ nparker@chromium.org, tsepez@chromium.org

[Lei Zhang, 2015-10-09 17:53:11]

I wrote this late last night and this is my solution for redirecting stdout +
stderr from worker processes back to the master. No idea if there's a better
way.

[Tom Sepez, 2015-10-09 19:41:59]

Looks OK, but I'll defer to the python gurus on this one.

[Lei Zhang, 2015-10-09 19:44:01]

On 2015/10/09 19:41:59, Tom Sepez wrote:
> Looks OK, but I'll defer to the python gurus on this one.

Yep, that's why I asked nparker to give it a shot.

BTW, on my workstation, it runs all the tests in parallel in 20 seconds, with
sane looking output.

[Nathan Parker, 2015-10-09 19:55:05]

I think this is a reasonable way to redirect.

https://codereview.chromium.org/1398793003/diff/1/testing/tools/common.py
File testing/tools/common.py (right):

https://codereview.chromium.org/1398793003/diff/1/testing/tools/common.py#new...
testing/tools/common.py:23: sys.stdout.write(subprocess.check_output(cmd,
stderr=subprocess.STDOUT))
Does the subprocess's stdout go to your modified sys.stdout, or do you want to
redirect that here as well?

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
File testing/tools/run_corpus_tests.py (right):

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
testing/tools/run_corpus_tests.py:49: def test_one_file_parallel(working_dir,
pdfium_test_path, image_differ, datum):
Nit: Some comments would be useful, especially for the types of non-descriptive
vars like datum.

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
testing/tools/run_corpus_tests.py:61: return (result, output.read(),
input_filename, source_dir)
I think .getvalue() works the same as .seek(0) + .read()

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
testing/tools/run_corpus_tests.py:112: worker_results = pool.imap(worker_func,
test_cases)
Do you want a timeout on these? Or is there a higher level deadline that'd kill
them all?

[Lei Zhang, 2015-10-09 20:14:33]

https://codereview.chromium.org/1398793003/diff/1/testing/tools/common.py
File testing/tools/common.py (right):

https://codereview.chromium.org/1398793003/diff/1/testing/tools/common.py#new...
testing/tools/common.py:23: sys.stdout.write(subprocess.check_output(cmd,
stderr=subprocess.STDOUT))
On 2015/10/09 19:55:04, nparker wrote:
> Does the subprocess's stdout go to your modified sys.stdout, or do you want to
> redirect that here as well? 

Yes, it seems to go to whatever sys.stdout is set to at the time this runs.

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
File testing/tools/run_corpus_tests.py (right):

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
testing/tools/run_corpus_tests.py:49: def test_one_file_parallel(working_dir,
pdfium_test_path, image_differ, datum):
On 2015/10/09 19:55:04, nparker wrote:
> Nit: Some comments would be useful, especially for the types of
non-descriptive
> vars like datum.

I added a brief comment and renamed datum to test_case. Let me know if that's
sufficient.

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
testing/tools/run_corpus_tests.py:61: return (result, output.read(),
input_filename, source_dir)
On 2015/10/09 19:55:04, nparker wrote:
> I think .getvalue() works the same as .seek(0) + .read()

Done.

https://codereview.chromium.org/1398793003/diff/1/testing/tools/run_corpus_te...
testing/tools/run_corpus_tests.py:112: worker_results = pool.imap(worker_func,
test_cases)
On 2015/10/09 19:55:04, nparker wrote:
> Do you want a timeout on these? Or is there a higher level deadline that'd
kill
> them all?

I believe buildbots have a timeout of its own. The serial path can also hang.

[Nathan Parker, 2015-10-09 20:32:45]

lgtm

[Lei Zhang, 2015-10-09 20:43:02]

Committed patchset #2 (id:20001) manually as
fd751f28cecce61ab36038799043639d570e0b26 (presubmit successful).