test.py: compute coverage per top-level module.

test.py

Index: test.py
diff --git a/test.py b/test.py
index bf199ce850fc315c49227ce84886567a0d25f814..6ddc5ac56ec02ef4952a427bb1f8ba739f3a2793 100755
--- a/test.py
+++ b/test.py
@@ -14,7 +14,7 @@ import sys
 
 
 def usage():
-  print """\nUsage: %s <action> [<expect_tests options>] [<test names>]
+  print """\nUsage: %s <action> [<test names>] [<expect_tests options>]

[pgervais, 2016/04/27 16:41:16]

The test/options order does not seem to matter. Is there a particular reason why
you inverted those?

[Sergey Berezin, 2016/04/27 17:32:14]

It actually does now, since I need to know the list of modules to test, and have
to assume that module names come first. The reason is, I cannot tell the
difference between a module and an argument to a flag, e.g. 

./test.py test infra_libs --html-report mydir

without a full fledged argparse, it's hard to tell that mydir is not a module.
So I simply collect non-flag arguments before the first flag, and call them
modules.

It's a bit of a breaking change, but the alternative of enforcing
--html-report=mydir format is also suboptimal, since I can't use ~/path and
expect it to work.

FWIW, in the original version, ./test.py test --html-report blah did NOT work as
expected (it didn't test all the modules), but now works well in the new
version.

[pgervais, 2016/04/27 17:40:00]

I'm afraid this is only the beginning of a long list of weird corner cases :-(
This test.py/expect_tests combo is really suboptimal.

 
   where <action> is one of: list, test, train, debug.
 
@@ -22,7 +22,7 @@ def usage():
   Run all tests:
     ./test.py test
   Run all tests in the infra package:
-    ./test.py test infra
+    ./test.py test infra --html-report /path/to/report/folder

[pgervais, 2016/04/27 16:41:16]

Is the --html-report option mandatory?

[Sergey Berezin, 2016/04/27 17:32:15]

No, I just added it as an example. I'll add another example without it, to be
clear.

   Run one given test in the infra package:
     ./test.py test infra/libs/git2/test:*testCommitBogus
 
@@ -69,23 +69,35 @@ else:
   python_bin = os.path.join('ENV', 'bin', 'python')
   expect_tests_path = os.path.join('ENV', 'bin', 'expect_tests')
 
-args = sys.argv[1:]
+command = sys.argv[1]
+args = sys.argv[2:]
+
+modules = []
+flags = []
+for arg in args:
+  if arg.startswith('-'):
+    flags.append(arg)
+    continue
+  if flags:
+    flags.append(arg)
+  else:
+    modules.append(arg)
 
 # Set up default list of packages/directories if none have been provided.
-if all([arg.startswith('--') for arg in sys.argv[2:]]):
+if not modules:
   if sys.platform == 'win32':
-    args.extend(WIN_ENABLED_PACKAGES)
+    modules.extend(WIN_ENABLED_PACKAGES)
   else:
-    args.extend(['infra', 'infra_libs'])  # TODO(pgervais): add 'test/'
+    modules.extend(['infra', 'infra_libs'])  # TODO(pgervais): add 'test/'
   appengine_dir = os.path.join(INFRA_ROOT, 'appengine')
   if sys.platform != 'win32' and os.path.isdir(appengine_dir):
-    args.extend(['appengine_module'])
+    modules.extend(['appengine_module'])
     appengine_dirs = [
       os.path.join('appengine', d)
       for d in os.listdir(appengine_dir)
     ]
     # Use relative paths to shorten the command-line
-    args.extend(itertools.chain(
+    modules.extend(itertools.chain(
       [d for d in appengine_dirs if os.path.isfile(os.path.join(d, 'app.yaml'))]
     ))
 
@@ -94,6 +106,17 @@ os.chdir(INFRA_ROOT)
 if '--help' not in sys.argv and '-h' not in sys.argv:

[pgervais, 2016/04/27 16:41:16]

'not in flags' instead?

[Sergey Berezin, 2016/04/27 17:32:15]

Done. I also improved printing the usage info.

   subprocess.check_call(
       [python_bin, os.path.join('bootstrap', 'remove_orphaned_pycs.py')])
-if sys.platform == 'win32' and '--force-coverage' not in args:
-  args.append('--no-coverage')
-sys.exit(subprocess.call([python_bin, expect_tests_path] + args))
+if sys.platform == 'win32' and '--force-coverage' not in flags:
+  flags.append('--no-coverage')
+
+exit_code = 0
+for module in modules:
+  print 'Running %s...' % module
+  module_flags = flags[:]
+  module_flags.append('--coveragerc=%s' % os.path.join(
+      INFRA_ROOT, module, '.coveragerc'))
+  module_flags.append('--html-report-subdir=%s' % module)
+  cmd = [python_bin, expect_tests_path] + [command] + [module] + module_flags
+  exit_code = max(exit_code, subprocess.call(cmd))

[pgervais, 2016/04/27 16:41:16]

'subprocess.call(cmd) or exit_code' instead

We want to return codes likes -1.

[Sergey Berezin, 2016/04/27 17:32:14]

Done.

Also printed the list of failing modules at the end of the run - otherwise
looking for one failure in the barrage of text is annoying.

+
+sys.exit(exit_code)