Subzero: Run cross tests as a much more configurable python script.

Subzero: Run cross tests as a much more configurable python script.

The runtests.sh script is removed and replaced with crosstest_generator.py.

"make check" limits to a relevant subset of cross tests to control the combinatorial explosion.  We cut the native tests almost in half, and the sandboxed tests down to a quarter.

The --include and --exclude logic is copied/adapted from szbuild.py.

The script works by running through every possible test in the combinatorial explosion, and if the test is a match against the --include and --exclude arguments, the test is built and run.

The script includes lit support, which is the most likely way it will be run.  When run with the --lit argument, it sprays the output directory with lit test files in the form of shell scripts, and "make check" runs lit on that directory.

BUG= https://code.google.com/p/nativeclient/issues/detail?id=4085
R=jvoung@chromium.org, mtrofin@chromium.org

Committed: https://gerrit.chromium.org/gerrit/gitweb?p=native_client/pnacl-subzero.git;a=commit;h=dc7c597ea93585ad9a56a06558260e4fac6d48ae

[Jim Stichnoth, 2015-03-06 08:17:05]

stichnot@chromium.org changed reviewers:
+ jvoung@chromium.org, kschimpf@google.com, mtrofin@chromium.org

[Jim Stichnoth, 2015-03-06 08:19:01]

This replaces runtests.sh with a much more flexible script.

A follow-on CL will adapt this script to auto-generate lit tests so that the
cross tests can be built and run in parallel, instead of sequentially as they
are today.

[Jim Stichnoth, 2015-03-07 00:40:12]

Really ready for review now.  I added lit support in this version.  The cross
tests take 1 minute to run on my laptop, and about 30 seconds on my workstation.

[Mircea Trofin, 2015-03-08 16:23:30]

https://codereview.chromium.org/987503004/diff/80001/crosstest/lit.cfg
File crosstest/lit.cfg (right):

https://codereview.chromium.org/987503004/diff/80001/crosstest/lit.cfg#newcode16
crosstest/lit.cfg:16: 'Release+Asserts/bin').format(root=FindBaseNaCl())
The path (llvm_x86_64_linux_work/Release+Asserts) means we can't run this on
windows or osx, nor on debug builds. Is this intentional?

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
File pydir/crosstest_generator.py (right):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
pydir/crosstest_generator.py:39: tests = {
will this list potentially grow? If so, is the expectation the developer edit
this file? In that case, could we simplify this by, for example, requiring a
naming pattern (like there appears to be one, too), and looking in a certain
directory for the tests?

[jvoung (off chromium), 2015-03-09 17:34:21]

https://codereview.chromium.org/987503004/diff/80001/Makefile.standalone
File Makefile.standalone (right):

https://codereview.chromium.org/987503004/diff/80001/Makefile.standalone#newc...
Makefile.standalone:232: $(LLVM_SRC_PATH)/utils/lit/lit.py -sv crosstest/Output
just checking what -sv means (succinct and verbose?)

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py
File pydir/crosstest.py (left):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py#oldco...
pydir/crosstest.py:18: build-pnacl-ir.py script.  The --prefix argument ensures
that
see question about build-pnacl-ir

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py#oldco...
pydir/crosstest.py:87: shellcmd(['../pydir/build-pnacl-ir.py',
'--disable-verify',
Is build-pnacl-ir still useful after this change?

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
File pydir/crosstest_generator.py (right):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
pydir/crosstest_generator.py:169: extra = tests[test]['flags'] if 'flags' in
tests[test] else []
Could probably have done "tests[test].get('flags', [])" ?

[Jim Stichnoth, 2015-03-09 18:16:17]

https://codereview.chromium.org/987503004/diff/80001/Makefile.standalone
File Makefile.standalone (right):

https://codereview.chromium.org/987503004/diff/80001/Makefile.standalone#newc...
Makefile.standalone:232: $(LLVM_SRC_PATH)/utils/lit/lit.py -sv crosstest/Output
On 2015/03/09 17:34:21, jvoung wrote:
> just checking what -sv means (succinct and verbose?)

Right, also note it was cargo-culted from the check-lit command.

https://codereview.chromium.org/987503004/diff/80001/crosstest/lit.cfg
File crosstest/lit.cfg (right):

https://codereview.chromium.org/987503004/diff/80001/crosstest/lit.cfg#newcode16
crosstest/lit.cfg:16: 'Release+Asserts/bin').format(root=FindBaseNaCl())
On 2015/03/08 16:23:30, Mircea Trofin wrote:
> The path (llvm_x86_64_linux_work/Release+Asserts) means we can't run this on
> windows or osx, nor on debug builds. Is this intentional?

Good point.  Changing this to be like the other lit test run, where this path is
passed in from Makefile.standalone where it can be overridden via an environment
variable.

Even that's not ideal and will need some work once we add new targets, but it's
tolerated because Makefile.standalone and lit tests are only developer tools,
not used by the bots.

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py
File pydir/crosstest.py (left):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py#oldco...
pydir/crosstest.py:18: build-pnacl-ir.py script.  The --prefix argument ensures
that
On 2015/03/09 17:34:21, jvoung wrote:
> see question about build-pnacl-ir

Ah, nice catch, updated the documentation.

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py#oldco...
pydir/crosstest.py:87: shellcmd(['../pydir/build-pnacl-ir.py',
'--disable-verify',
On 2015/03/09 17:34:21, jvoung wrote:
> Is build-pnacl-ir still useful after this change?

It isn't used by our scripts, but it's still kind of useful for
hand-constructing a lit test or a .ll cross test starting from C/C++ source. 
What do you think?

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
File pydir/crosstest_generator.py (right):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
pydir/crosstest_generator.py:39: tests = {
On 2015/03/08 16:23:30, Mircea Trofin wrote:
> will this list potentially grow? If so, is the expectation the developer edit
> this file? In that case, could we simplify this by, for example, requiring a
> naming pattern (like there appears to be one, too), and looking in a certain
> directory for the tests?

Good point that I should try not to hard-code the tests into the driver script.

I'd prefer not to impose file naming conventions on the tests, and there's also
the issue where test_sync_atomic needs additional flags.  Instead, I essentially
moved this dictionary into a separate config file.

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
pydir/crosstest_generator.py:169: extra = tests[test]['flags'] if 'flags' in
tests[test] else []
On 2015/03/09 17:34:21, jvoung wrote:
> Could probably have done "tests[test].get('flags', [])" ?

Hmm, nice.  Unfortunately, I don't think there's an equivalent for the new
ConfigParser version of this?

[jvoung (off chromium), 2015-03-09 20:34:59]

lgtm

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py
File pydir/crosstest.py (left):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py#oldco...
pydir/crosstest.py:87: shellcmd(['../pydir/build-pnacl-ir.py',
'--disable-verify',
On 2015/03/09 18:16:16, stichnot wrote:
> On 2015/03/09 17:34:21, jvoung wrote:
> > Is build-pnacl-ir still useful after this change?
> 
> It isn't used by our scripts, but it's still kind of useful for
> hand-constructing a lit test or a .ll cross test starting from C/C++ source. 
> What do you think?


Hmm, up to you. I guess it could be a useful shortcut for that. I hadn't thought
to use it that way to start a .ll file.

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
File pydir/crosstest_generator.py (right):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest_generato...
pydir/crosstest_generator.py:169: extra = tests[test]['flags'] if 'flags' in
tests[test] else []
On 2015/03/09 18:16:16, stichnot wrote:
> On 2015/03/09 17:34:21, jvoung wrote:
> > Could probably have done "tests[test].get('flags', [])" ?
> 
> Hmm, nice.  Unfortunately, I don't think there's an equivalent for the new
> ConfigParser version of this?

Hmm I don't see anything like that for ConfigParser, so this is fine =)

[Jim Stichnoth, 2015-03-09 20:37:48]

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py
File pydir/crosstest.py (left):

https://codereview.chromium.org/987503004/diff/80001/pydir/crosstest.py#oldco...
pydir/crosstest.py:87: shellcmd(['../pydir/build-pnacl-ir.py',
'--disable-verify',
On 2015/03/09 20:34:59, jvoung wrote:
> On 2015/03/09 18:16:16, stichnot wrote:
> > On 2015/03/09 17:34:21, jvoung wrote:
> > > Is build-pnacl-ir still useful after this change?
> > 
> > It isn't used by our scripts, but it's still kind of useful for
> > hand-constructing a lit test or a .ll cross test starting from C/C++ source.

> > What do you think?
> 
> 
> Hmm, up to you. I guess it could be a useful shortcut for that. I hadn't
thought
> to use it that way to start a .ll file.

Heh, that's where all those .pnacl.ll lit tests (and one cross test) came from.
:)

[Mircea Trofin, 2015-03-10 17:12:23]

lgtm

[Jim Stichnoth, 2015-03-10 18:17:20]

Committed patchset #7 (id:120001) manually as
dc7c597ea93585ad9a56a06558260e4fac6d48ae (presubmit successful).