Test infrastructure for end-to-end tests of GYP generators, and...

Test infrastructure for end-to-end tests of GYP generators, and
initial tests of building a simple "Hello, world!" executable.

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=562

[sgk, 2009-07-29 21:04:39]

This is for preliminary comments.

The goal here is to use live tools to test the actual *functionality* that we
expect from the generated build configurations.  That is, instead of (e.g.)
examining generated Makefiles for exact matches, we run Make to verify that the
gyp-generated build config does what it's supposed to.  IMHO, we should be able
to refactor a generator to complete redesign the generated build config files so
long as the generated build continues to function correctly.

Test configurations would be one per subdirectory tree; that is, test/hello in
this CL represents one testable configuration.  Individual test scripts that use
a configuration would be named 'gyptest*.py' within the configuration's
directory.  This CL has three simple tests of a "Hello, world!" build; take a
look at test/hello/gyptest-*.py.

test/lib/Test{Cmd,Common}.py originate outside the project (I originally wrote
them for SCons, but they're generic) and don't need thorough review (although
comments are always welcome).

test/lib/TestGyp.py is the gyp test infrastructure that needs some scrutiny so
that the general structure and "feel" are something everyone is comfortable
with.

Ignore the test/hello/site_scons subdirectory.  That's copied from the current
chromium build because the SCons backend is (incorrectly) chromium-specific.  A
follow-on CL might fix the SCons generator to make the relevant functionality in
chromium_builders.py part of its generated SCons files.  (Finishing the Makefile
generator and making it official would make that moot, of course.)

This also still needs a wrapper script for simpler control of test execution. 
Right now I'm feeding scripts to Python and setting environment vaiables by
hand, grungy stuff that's better wrapped up in a script.  I wanted to get a
thumbs up on the general direction before putting additional time into that,
though.

That said, here's output from running the tests with XCode:

$ PYTHONPATH=`pwd`/test/lib TESTGYP_FORMAT=xcode python
test/hello/gyptest-all.py
PASSED
$ PYTHONPATH=`pwd`/test/lib TESTGYP_FORMAT=xcode python
test/hello/gyptest-default.py
PASSED
$ PYTHONPATH=`pwd`/test/lib TESTGYP_FORMAT=xcode python
test/hello/gyptest-target.py
PASSED
$

All three tests pass using msvs, xcode and scons.  The make generator has two
behavioral differences that may or may not be relevant:  1) 'make hello' gives
me compilation errors while 'make all' seems to work; 2) 'all' does not seem to
be the default target in the generated Makefeil, so you don't build the 'hello'
executable by default.

If this infrastructure looks like something we're all okay with, I would finish
the pieces I've mentioned above, start adding additional tests, and also start
insisting that all future changes to the generators come with new or modified
test cases.

Fire away...  :-)

[sgk, 2009-07-30 00:08:39]

[Re-sending separately to correctly spelled gyp-developer group name.  --SK]

This is for preliminary comments.

The goal here is to use live tools to test the actual *functionality* that we
expect from the generated build configurations.  That is, instead of (e.g.)
examining generated Makefiles for exact matches, we run Make to verify that the
gyp-generated build config does what it's supposed to.  IMHO, we should be able
to refactor a generator to complete redesign the generated build config files so
long as the generated build continues to function correctly.

Test configurations would be one per subdirectory tree; that is, test/hello in
this CL represents one testable configuration.  Individual test scripts that use
a configuration would be named 'gyptest*.py' within the configuration's
directory.  This CL has three simple tests of a "Hello, world!" build; take a
look at test/hello/gyptest-*.py.

test/lib/Test{Cmd,Common}.py originate outside the project (I originally wrote
them for SCons, but they're generic) and don't need thorough review (although
comments are always welcome).

test/lib/TestGyp.py is the gyp test infrastructure that needs some scrutiny so
that the general structure and "feel" are something everyone is comfortable
with.

Ignore the test/hello/site_scons subdirectory.  That's copied from the current
chromium build because the SCons backend is (incorrectly) chromium-specific.  A
follow-on CL might fix the SCons generator to make the relevant functionality in
chromium_builders.py part of its generated SCons files.  (Finishing the Makefile
generator and making it official would make that moot, of course.)

This also still needs a wrapper script for simpler control of test execution. 
Right now I'm feeding scripts to Python and setting environment vaiables by
hand, grungy stuff that's better wrapped up in a script.  I wanted to get a
thumbs up on the general direction before putting additional time into that,
though.

That said, here's output from running the tests with XCode:

$ PYTHONPATH=`pwd`/test/lib TESTGYP_FORMAT=xcode python
test/hello/gyptest-all.py
PASSED
$ PYTHONPATH=`pwd`/test/lib TESTGYP_FORMAT=xcode python
test/hello/gyptest-default.py
PASSED
$ PYTHONPATH=`pwd`/test/lib TESTGYP_FORMAT=xcode python
test/hello/gyptest-target.py
PASSED
$

All three tests pass using msvs, xcode and scons.  The make generator has two
behavioral differences that may or may not be relevant:  1) 'make hello' gives
me compilation errors while 'make all' seems to work; 2) 'all' does not seem to
be the default target in the generated Makefeil, so you don't build the 'hello'
executable by default.

If this infrastructure looks like something we're all okay with, I would finish
the pieces I've mentioned above, start adding additional tests, and also start
insisting that all future changes to the generators come with new or modified
test cases.

Fire away...  :-)

[Greg Spencer, 2009-07-30 00:25:48]

Cool!  I just proposed adding a simple test for the variable expansions from
input.py in another CL (http://codereview.chromium.org/159605 -- I made you a
reviewer).

I'm guessing you'd rather I use this (and I'd rather use this!).  Will this
easily test basic GYP parsing functionality as well (input.py)?

[Mark Mentovai, 2009-07-30 16:26:50]

I like this approach.

http://codereview.chromium.org/160351/diff/1/8
File trunk/test/hello/gyptest-all.py (right):

http://codereview.chromium.org/160351/diff/1/8#newcode16
Line 16: test.run_built_executable('hello', stdout="Hello, world!\n")
Oh, this is a good idea.

http://codereview.chromium.org/160351/diff/1/4
File trunk/test/hello/hello.c (right):

http://codereview.chromium.org/160351/diff/1/4#newcode1
Line 1: #include <stdio.h>
Maybe make this match Google style a little bit better (just formatting,
really).

http://codereview.chromium.org/160351/diff/1/6
File trunk/test/hello/site_scons/site_tools/chromium_builders.py (right):

http://codereview.chromium.org/160351/diff/1/6#newcode1
Line 1: # Copyright (c) 2006-2008 The Chromium Authors. All rights reserved.
Is this file a copy of something from Chromium?  Or a modification of something
from Chromium?  Looks like it, so it'd be good to include a comment saying where
it comes from.

http://codereview.chromium.org/160351/diff/1/11
File trunk/test/lib/TestCmd.py (right):

http://codereview.chromium.org/160351/diff/1/11#newcode1
Line 1: """
README in this directory so we know where this came from.

http://codereview.chromium.org/160351/diff/1/10
File trunk/test/lib/TestGyp.py (right):

http://codereview.chromium.org/160351/diff/1/10#newcode213
Line 213: class TestGypXCode(TestGypBase):
nit: Xcode is capital X, lowercase c

[sgk, 2009-07-30 21:10:47]

Added gyptest.py to control execution, and test/lib/README.txt.  Reorganized
test/lib/TestGyp.py a bit to add docstrings and collect the abstract methods in
one location.  Other fixes based on feedback.

http://codereview.chromium.org/160351/diff/1/4
File trunk/test/hello/hello.c (right):

http://codereview.chromium.org/160351/diff/1/4#newcode1
Line 1: #include <stdio.h>
On 2009/07/30 16:26:50, Mark Mentovai wrote:
> Maybe make this match Google style a little bit better (just formatting,
> really).

I changed the indent to four-wide spaces.  LMK if there are other formatting
things you had in mind.

http://codereview.chromium.org/160351/diff/1/6
File trunk/test/hello/site_scons/site_tools/chromium_builders.py (right):

http://codereview.chromium.org/160351/diff/1/6#newcode1
Line 1: # Copyright (c) 2006-2008 The Chromium Authors. All rights reserved.
On 2009/07/30 16:26:50, Mark Mentovai wrote:
> Is this file a copy of something from Chromium?  Or a modification of
something
> from Chromium?  Looks like it, so it'd be good to include a comment saying
where
> it comes from.

This whole file is just cut-and-paste until I can fix the SCons generator to get
rid of it.  The generator makes use of this functionality I had added to the
Chromium build, and it should really be atomic and generate the info directly. 
I'll add a comment just for clarification, though.

http://codereview.chromium.org/160351/diff/1/11
File trunk/test/lib/TestCmd.py (right):

http://codereview.chromium.org/160351/diff/1/11#newcode1
Line 1: """
On 2009/07/30 16:26:50, Mark Mentovai wrote:
> README in this directory so we know where this came from.

Done.

http://codereview.chromium.org/160351/diff/1/10
File trunk/test/lib/TestGyp.py (right):

http://codereview.chromium.org/160351/diff/1/10#newcode213
Line 213: class TestGypXCode(TestGypBase):
On 2009/07/30 16:26:50, Mark Mentovai wrote:
> nit: Xcode is capital X, lowercase c

Done.

[Mark Mentovai, 2009-07-30 21:20:26]

Just addressing the C style bit here.

http://codereview.chromium.org/160351/diff/38/41
File trunk/test/hello/hello.c (right):

http://codereview.chromium.org/160351/diff/38/41#newcode4
Line 4: int
int main on one line (per style)

http://codereview.chromium.org/160351/diff/38/41#newcode7
Line 7: printf("Hello, world!\n");
C/C++ use a 2-space indent (per style)

http://codereview.chromium.org/160351/diff/38/41#newcode8
Line 8: exit(0);
Personally, I "return 0;" from main instead of calling exit.  Some compilers
actually give a hoot.  You can drop stdlib if you change this to return.