Add Python flow to bindings generation, move dummy-generating IDL files over

Source/bindings/scripts/build_ir.py

+# Copyright (C) 2013 Google Inc. All rights reserved.

[haraken, 2013/07/08 06:06:29]

Actually I don't see a benefit in splitting build_ir.py and
generate-bindings.py. Shall we put the contents of these two files into one file
and call it idl_compiler.py? Then the architecture becomes clearer:

  idl_compiler.py : driver
  idl_parser.py : parser
  code_generator_v8.py : code generator

[Nils Barth (inactive), 2013/07/09 08:48:16]

In the context of the full parsing, these are quite separate,
though admittedly it's not clear just from this initial CL.

build_ir.py is actually a major file (700 lines in local branch);
this is just a stub.
You can look at the full file here:
https://codereview.chromium.org/15801003/

Agree that the current file is pretty small, but it's the start of a big
library.

build_ir.py does three things:
1. Convert the AST produced by the parser into the IR that the code generator
uses. This is the bulk of the work.
2. Merge dependencies into the main interface.
3. Validates IDL extended attributes.

Once these are all done, we have a complete, validated IR that can
be passed on to the code generator.

Architecturally, read_merged_idl is the main function, which
reads in an IDL file and does all the necessary work.
It's important that this be in a separate library from our compiler,
since other people (notably Dart, and in future bindings) want to
be able to read IDL files using our code, so they need to be able
to call the merging and validation code.
idl_compiler.py is a clearer name than generate_bindings.py;
changed!
This actually is the architecture, but the "parser" is actually
in 3 stages, hence why I'm referring to it as "frontend":
* blink_idl_lexer.py : lexer (makes token stream)
* blink_idl_parser.py : parser (makes AST)
* build_ir.py : converts AST to IR, merges dependencies, validates attributes
(we need the "blink_" prefixes b/c the underlying Pepper parser uses
idl_lexer.py and idl_parser.py itself).

So the architecture is:
* driver
* frontend: (lexer, parser, AST to IR + etc.)
* backend: code generator
...as you suggest!

(We don't need to use the term "backend" here, since we only have one, but Dart
uses 3 backends, for instance, and extensions may use another one.)

+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Convert generic Web IDL AST to Blink IR.

[haraken, 2013/07/08 06:06:29]

As commented below, I'd avoid using the word "IR" since the meaning is
ambiguous. You can use "parsing tree" instead.

[Nils Barth (inactive), 2013/07/09 08:48:16]

The code generator takes as input a Python object (of class IdlDocument), which
is our IR. So the term "IR" is unambiguous, but unclear without more context –
I've clarified the comments.
It's clear when you look at ir.py (in the full parser CL), which shows
the full class hierarchy, but it should be clear from this comment as well.

The term "parsing tree" (Concrete Syntax Tree, CST, corresponding 
directly to the production rules of the grammar) isn't accurate.

Rather:
The parser itself (blink_idl_parser.py) walks the CST and produces an AST
(discarding useless grammar rules, building lists, etc.).
It's then a lot of code to convert this AST to the Python object,
which is most of what build_ir.py actually does (eventually, once it lands).

That's why I'm using the term "IR": we've processed the tree and produced
something actually usable, so the code generator doesn't need to
walk the tree itself.
(This architecture is following the existing Perl: make an object.)

+
+This is the last phase of the frontend, after the lexer and the parser.
+This also merges interface dependencies (partial interfaces and implements),
+and validates extended attributes.

[haraken, 2013/07/08 06:06:29]

Where is this implemented?

[Nils Barth (inactive), 2013/07/09 08:48:16]

The validation isn't included in this CL, which is just the minimum
necessary for a testable build change.
The conversion of the AST to the IR also isn't included.

The comment is "what this library will eventually include",
and there's a "FIXME: Currently a stub" comment below.
Should I make that more prominent (put the FIXME first?),
or add a comment like "NOT YET LANDED", or otherwise reword?

+The IR is then used by the code generator to produce .cpp/.h files.
+
+Ideally the parser would generate the IR directly, rather then requiring this
+additional phase, but that would require major changes to the parser, which
+currently produces a generic AST instead.
+
+FIXME: Currently a stub, as part of landing the parser incrementally.
+Just computes supplemental files, and should always raise an exception,
+indicating no file found, so no actual parsing done.
+"""
+
+import os.path
+
+# Generic Web IDL AST to Blink IR

[haraken, 2013/07/08 06:06:29]

Nit: Remove this comment.

[Nils Barth (inactive), 2013/07/09 08:48:16]

Done.
(The full file is rather long, so I have comments to break up
the sections, but certainly now these are excessive.)

+
+
+def read_merged_idl(idl_filename, supplemental_dependency_file=None, additional_idl_files=None):

[haraken, 2013/07/08 06:06:29]

read_merged_idl => resolve_idl_dependency ?

[Nils Barth (inactive), 2013/07/09 08:48:16]

This actually calls the parser, converts to IR, does merging, and validation –
it's the major function – but I've currently removed all of these, so all that's
left is resolving the dependencies.
It's probably clearer if I include these other actions but commented out: see
revised CL.

+    """Returns Blink IR for an IDL file, including all dependencies."""
+    basename = os.path.basename(idl_filename)
+
+    # First check supplemental files, so throws exception if not listed.
+    supplemental_idl_files = compute_supplemental_idl_files(basename, supplemental_dependency_file, additional_idl_files)
+    # FIXME: currently *must* raise IdlNotFoundError (indicating that dummy
+    # .cpp and .h files should be generated), as actual parser not present yet.
+    raise RuntimeError('Stub: parser not implemented yet')
+
+# Merge supplemental interfaces ('partial interface' and 'implements')

[haraken, 2013/07/08 06:06:29]

Nit: Remove this comment.

[Nils Barth (inactive), 2013/07/09 08:48:16]

Done.

+
+
+class IdlNotFoundError(Exception):
+    """Raised if IDL not found in dependencies file or additional files list."""

[haraken, 2013/07/08 06:06:29]

Slightly better:

  Raised if the IDL file is a partial IDL file.

[Nils Barth (inactive), 2013/07/09 08:48:16]

Dependencies now include both partial IDL files and implements,
neither of which we generate bindings for.
It is unclear though; I've reworded it for clarity.

+    pass
+
+
+def compute_supplemental_idl_files(target_idl_basename, supplemental_dependencies_filename, additional_idl_files):

[haraken, 2013/07/08 06:06:29]

compute_supplemental_idl_files => compute_partial_idl_files
supplemental_dependencies_filename => partial_dependencies_filename

[Nils Barth (inactive), 2013/07/09 08:48:16]

Changed!
(Actually to "dependencies" because also includes 'implements'.)

+    """Returns list of IDL file dependencies for a given main IDL file.
+
+    Also returns empty list if input is an additional IDL file (one not part
+    of main bindings).
+    If an IDL file is neither in the main bindings (listed in the dependencies
+    file) nor in the additional IDL files list, raise an IdlNotFoundError.

[haraken, 2013/07/08 06:06:29]

Slightly better:

- Returns a list of partial IDL files on which a given IDL file depends.
- Returns an empty list if the given IDL file is an additional IDL file.
- Raises IdlNotFoundError otherwise. This happens when the given IDL file is a
partial IDL file, for which we don't want to generate .h and .cpp files.

[Nils Barth (inactive), 2013/07/09 08:48:16]

I've reworded following this, trying to avoid redundancy with the
description of the exception.
We actually *do* want to generate .h and .cpp files for the dependencies,
but they're supposed to be dummy ones.
I've said "we don't want to generate bindings", though we could say
"don't want to generate *real* .h and .cpp files" (which sounds a bit
more confusing to me).

+    """
+    # The format of the supplemental dependencies file is:
+    #
+    # DOMWindow.idl P.idl Q.idl R.idl
+    # Document.idl S.idl
+    # Event.idl
+    # ...
+    #
+    # The above indicates that:
+    # DOMWindow.idl is supplemented by P.idl, Q.idl, and R.idl,

[haraken, 2013/07/08 06:06:29]

P.idl, Q.idl and R.idl are partial files of DOMWindow.idl.

The word "supplemental" is deprecated. Please replace it with "partial" in all
other parts in this CL.

[Nils Barth (inactive), 2013/07/09 08:48:16]

Replaced with "dependencies", since includes both partials and implements.

Done, and fixed everywhere in other CL:
Issue 18341025: Rename deprecated "supplemental_dependencies" and clean up IDL
dependency script
https://codereview.chromium.org/18341025/

+    # Document.idl is supplemented by S.idl, and
+    # Event.idl is supplemented by no IDL files.
+    #
+    # An IDL file that supplements another IDL file (e.g., P.idl in the above)
+    # does not appear on its own line in the dependency file: supplemental
+    # files are not themselves supplemented.
+    if supplemental_dependencies_filename is None:
+        return []

[haraken, 2013/07/08 06:06:29]

This should raise an exception.

[Nils Barth (inactive), 2013/07/09 08:48:16]

The dependencies file is optional, as is the list of additional IDL files
(defaults to empty).
(They're both currently optional in Perl.)
We could change this and make the dependencies file required.
It's slightly cleaner to be optional, as it means we don't depend on
compute_dependencies.py unless we have to (so you can compile a single
IDL file).

+    with open(supplemental_dependencies_filename) as supplemental_dependencies_file:
+        for line in supplemental_dependencies_file:
+            idl_filename, _, dependency_files = line.partition(' ')
+            if os.path.basename(idl_filename) == target_idl_basename:
+                return dependency_files.split()
+
+    # additional_idl_files is a list of IDL files which should not be included
+    # in DerivedSources*.cpp, and hence are not listed in the supplemental
+    # dependency file, but for which we should generate .cpp and .h files.
+    additional_idl_basenames = set(map(os.path.basename, additional_idl_files))
+    if target_idl_basename in additional_idl_basenames:
+        return []
+    raise IdlNotFoundError