Add support for khronos gl-cts using its drawElements apis

gpu/khronos_conform_support/generate_khronos_conform_tests.py

+#!/usr/bin/env python

[piman, 2014/09/10 20:12:50]

This file looks very similar to 
gpu/gles2_conform_support/generate_gles2_conform_tests.py. Can we share / avoid
duplication?

[U. Artie Eoff, 2014/09/10 22:12:59]

There are quite a few differences between the two test generators.  The new
drawElements framework uses a different test naming scheme and this accounts for
that.  Also, this generator recurses through the .run files and only produces
tests for the .test files (the algorithm is similar to what can be found in the
drawElements framework when it traverses the .run files to produce test names). 
The generate_gles2_conform_tests.py does not recurse.

I suppose the generate_gles2_conform_tests.py could be refactored to support
both schemes... but I think it would likely get a little messy.

[Ken Russell (switch to Gerrit), 2014/09/11 00:03:26]

As unfortunate as code duplication in this area is, I can understand the need to
do it because Khronos' test harness has evolved dramatically between the GLES2
and GLES3 conformance suites.

Can this new test suite subsume the GLES2 conformance tests? Are all of the
areas covered by the GLES2 conformance tests also covered by these new tests?

[U. Artie Eoff, 2014/09/11 01:09:20]

Yes, this new suite *does* subsume the ES2 conformance tests.  In fact, the ES2
tests are the only tests this patch "subsumes" at the moment.  The pudding in
the cake is (using the newer framework/APIs) we can extend this integration
later to include the ES3+ tests, trivially.  So, yes, all of the areas covered
by the GLES2 conformance tests *are* covered here, and then some (i.e. presuming
there have been more es2 test additions and fixes since the "legacy" khronos
suite).

[Ken Russell (switch to Gerrit), 2014/09/11 01:25:17]

That's very good. We would be highly interested in getting this new target
running on the http://build.chromium.org/p/chromium.gpu.fyi/console waterfall
and removing the gles2_conform_test target in that case. Thank you for
confirming this.

What platforms have you tested this on? Have you at least tested it on desktop
Linux as well as Chrome OS? If we integrate the GLES3 conformance test snapshot
in src-internal, do you expect this will work fairly easily on the platforms
that Chromium's port of the GLES2 conformance test runs on?

[U. Artie Eoff, 2014/09/11 16:50:53]

I haven't been able to get *pristine* Chromium to compile successfully on
desktop Linux, yet.  Granted, I haven't spent much time trying to figure out
why.

The only platform I've tested this CL on is ChromiumOS.

I expect there will be a little enabling work from your end to ensure this ran
on other platforms.  I don't have access to all those platforms, nor what that
list of platforms is.  I assume MAC and Windows are on that list.  I could
extrapolate details from the GLES2 conformance support, but I would be
uncomfortable doing that without being able to test it properly.

Nonetheless, I imagine the enabling of other platforms should be fairly easy. 
The nice thing is it can be done incrementally without affecting the legacy
GLES2 conformance support.

Mainly, I'm asking for your support in this realm.

[Ken Russell (switch to Gerrit), 2014/09/11 17:40:04]

You certainly have my support. Could you please update this CL with the AUTHORS
change piman requested and the other naming convention changes? I'll help you
get the Khronos files checked in to the appropriate place. May we rely on you to
help advise on issues we may encounter porting this to the other platforms?

[U. Artie Eoff, 2014/09/11 19:45:25]

Excellent! The CL updates are on the way ;) Yes, I will help advise on any
issues encountered.

+# Copyright 2014 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Code generator for khronos_conformance tests."""
+
+import os
+import re
+import sys
+import argparse
+
+TEST_DEF_TEMPLATE = """
+TEST(KhronosConformTest, %(gname)s) {
+  EXPECT_TRUE(RunKhronosConformTest("%(cname)s"));
+}
+"""
+
+RUN_FILE_SUITE_PREFIX = {
+  "mustpass_es20.run" : "ES2-CTS.gtf",
+}
+
+def ReadFileAsLines(filename):
+  """
+    Reads a file, yielding each non-blank line
+    and lines that don't begin with #
+  """
+  file = open(filename, "r")
+  lines = file.readlines()
+  file.close()
+  for line in lines:
+    line = line.strip()
+    if len(line) > 0 and not line.startswith("#"):
+      yield line
+
+def ReadRunFile(run_file):
+  """
+    Find all .test tests in a .run file and return their paths.
+    If the .run file contains another .run file, then that is inspected
+    too.
+  """
+  tests = list()
+  base_dir = os.path.dirname(run_file)
+  for line in ReadFileAsLines(run_file):
+    root, ext = os.path.splitext(line)
+    if ext == ".test":
+      tests.append(os.path.join(base_dir, line))
+    elif ext == ".run":
+      tests += ReadRunFile(os.path.join(base_dir, line))
+    else:
+      raise ValueError, "Unexpected line '%s' in '%s'" % (line, run_file)
+  return tests
+
+def GenerateTests(run_files, output):
+  """
+    Generates code for khronos_conform_test test-cases that are
+    listed in the run_files.
+  """
+  output.write('#include "gpu/khronos_conform_support/khronos_conform_test.h"\n')
+  output.write('#include "testing/gtest/include/gtest/gtest.h"\n\n')
+
+  for run_file in run_files:
+    run_file_name = os.path.basename(run_file)
+    run_file_dir = os.path.dirname(run_file)
+    suite_prefix = RUN_FILE_SUITE_PREFIX[run_file_name]
+    output.write("// " + run_file_name + "\n")
+    for test in ReadRunFile(run_file):
+      rel_path = os.path.relpath(test, run_file_dir)
+      root, ext = os.path.splitext(rel_path)
+      name = root.replace('.', '_')
+      name = "%s.%s" % (suite_prefix, name.replace(os.path.sep, '.'))
+      output.write(TEST_DEF_TEMPLATE
+        % {
+          "gname": re.sub(r'[^A-Za-z0-9]', '_', name),
+          "cname": name,
+        })
+    output.write("\n");
+
+def main():
+  """This is the main function."""
+  parser = argparse.ArgumentParser()
+  parser.add_argument("--outdir", default = ".")
+  parser.add_argument("run_files", nargs = "+")
+
+  args = parser.parse_args()
+
+  output = open(
+    os.path.join(args.outdir, "khronos_conform_test_autogen.cc"), "wb")
+
+  try:
+    GenerateTests(args.run_files, output)
+  finally:
+    output.close()
+
+  return 0
+
+if __name__ == '__main__':
+  sys.exit(main())