Add Python flow to bindings generation, move dummy-generating IDL files over

Source/bindings/scripts/read_idl.py

+# Copyright (C) 2013 Google Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Read an IDL file, producing an IdlDocument object.
+
+This library calls the parser, which produces a generic Web IDL AST.
+The AST is then converted into the IR, a Python object of class IdlDocument.
+This is the last phase of the frontend, after the lexer and the parser.
+
+It also provides functions to merges interface dependencies (partial interfaces
+and implements), and to validate extended attributes.
+After calling these functions, the result is an IdlDocument representing a
+complete, validated interface.
+The IdlDocument is then used by the code generator to produce .h/.cpp files.
+
+Ideally the parser would generate the IdlDocument directly, rather then
+requiring this additional phase, but that would require major changes to the
+parser, which currently produces a generic AST instead.
+
+FIXME: Currently a stub, as part of landing the parser incrementally.
+Just computes dependencies, and should always raise an exception,
+indicating no file found, so no actual parsing done.
+"""
+
+import os.path
+
+
+class IdlDocument():
+    """Top-level IDL class.
+
+    In Web IDL spec terms, represents a set of definitions, obtained by parsing
+    a set of IDL fragments (i.e., the contents of .idl files)."""
+    # FIXME: Dummy class; full class hierarchy will be added with parser
+    pass
+
+
+def read_idl_file(idl_filename, verbose=False):
+    """Returns an IdlDocument for an IDL file, without any dependencies."""
+    # FIXME: Currently returns dummy object, as parser not present yet.
+    # parser = BlinkIDLParser(verbose=verbose)
+    # file_node = ParseFile(parser, idl_filename)
+    # return file_node_to_idl_document(file_node)
+    return IdlDocument()
+
+
+def merge_interface_dependencies(idl_document, idl_filename, interface_dependencies_file, additional_idl_files):
+    """Merges dependencies into an existing IDL document.
+
+    Modifies idl_document in place by adding parsed dependencies, and checks
+    whether bindings should be generated, returning bool.
+
+    FIXME: stub: parser not implemented yet
+
+    Arguments:
+        idl_document: IdlDocument object, modified in place
+        idl_filename: filename of main IDL file

[haraken, 2013/07/10 04:06:50]

Nit: idl_filename => idl_file (for consistency with other parameter names)

BTW, how about passing an interface name to this method instead of a file name?

[Nils Barth (inactive), 2013/07/10 06:00:15]

Ack, I've been using _file for the File object itself, and _filename for the
string,
but it's inconsistent. I've made it consistent.
That's a good clean-up, and it's on my list.
The problem is that dependencies are keyed by *full path*, not interface.
We can fix this by keying these by interface, but that requires changing
several scripts.
I'll do this in a followup.

+        interface_dependencies_file: filename of dependencies file (produced by compute_dependencies.py)
+        additional_idl_files: list of additional files, not listed in interface_dependencies_file, for which bindings should nonetheless be generated
+    Returns:
+        bool, whether bindings should be generated or not.
+    """
+    basename = os.path.basename(idl_filename)
+    interface_name, _ = os.path.splitext(basename)
+
+    dependency_idl_files = compute_dependency_idl_files(basename, interface_dependencies_file, additional_idl_files)
+    if dependency_idl_files is None:
+        return False
+    # FIXME: currently dependency_idl_files *must* be None (indicating that
+    # dummy .cpp and .h files should be generated), as actual parser not
+    # present yet.
+    raise RuntimeError('Stub: parser not implemented yet')
+
+
+def compute_dependency_idl_files(target_idl_basename, interface_dependencies_filename, additional_idl_files):

[haraken, 2013/07/10 04:06:50]

compute_dependency_idl_files => compute_interface_dependency ?

[Nils Barth (inactive), 2013/07/10 06:00:15]

This actually returns the file names:
since partial interfaces don't have distinct interface names,
we need the actual file names.
(See details below.)

+    """Returns list of IDL file dependencies for a given main IDL file.
+
+    - Returns a list of IDL files on which a given IDL file depends,
+      possibly empty.
+      Dependencies consist of partial interface files and files for other
+      interfaces that the given interface implements.
+    - Returns an empty list also if the given IDL file is an additional IDL
+      file.
+    - Otherwise, return None. This happens when the given IDL file is a
+      dependency, for which we don't want to generate bindings.
+    """
+    if interface_dependencies_filename is None:
+        return []
+
+    # The format of the interface dependencies file is:
+    #
+    # Document.idl P.idl
+    # Event.idl
+    # Window.idl Q.idl R.idl S.idl

[haraken, 2013/07/10 04:06:50]

Probably the format of the dependency file should be:

  Document P
  Event
  Window Q R S

because what we want to describe here is not dependencies of IDL files but
dependencies of interfaces.

You can fix it before or after this CL.

[Nils Barth (inactive), 2013/07/10 06:00:15]

Yeah, this definitely needs cleaning up, but it's a bit messy.

The dependencies include both partial interfaces, which only have a
filename (the interface name is the same as for the main file),
*and* 'implements' statements, for which the interface name is enough,
but for which we're storing the full path.
This also is used to lookup full paths, given an interface name.
It's really a mess.

What we really want is a dictionary mapping:
interface_name -> main_idl_filename, [partial_idl_filenames],
[implemented_interface_names]

This should be generated by compute_dependencies.py,
and would make a lot of code a lot clearer, since we wouldn't
have to keep passing IDL filenames around.
The only time we actually need a filename is when reading a file,
but the full path is currently used as a key several places.
This requires fixing a few other scripts, so I'll do it after
(better to land this and the parser first, since those are blocking,
then do cleanup).

+    # ...
+    #
+    # The above indicates that:
+    # Document.idl depends on P.idl,
+    # Event.idl depends on no other IDL files, and
+    # Window.idl depends on Q.idl, R.idl, and S.idl.
+    #
+    # The head entries (first IDL file on a line) are the files that should
+    # have bindings generated.
+    #
+    # An IDL file that is a dependency of another IDL file (e.g., P.idl in the
+    # above example) does not have its own line in the dependency file:
+    # dependencies do not have bindings generated, and do not have their
+    # own dependencies.
+    with open(interface_dependencies_filename) as interface_dependencies_file:
+        for line in interface_dependencies_file:
+            idl_filename, _, dependency_files = line.partition(' ')
+            if os.path.basename(idl_filename) == target_idl_basename:
+                return dependency_files.split()
+
+    # additional_idl_files is a list of IDL files which should not be included
+    # in DerivedSources*.cpp, and hence are not listed in the interface
+    # dependencies file, but for which we should generate .cpp and .h files.
+    additional_idl_basenames = set(map(os.path.basename, additional_idl_files))
+    if target_idl_basename in additional_idl_basenames:
+        return []
+
+    return None