IDL parser rewrite in Python

Source/bindings/scripts/ir.py

+# Copyright (C) 2013 Google Inc. All rights reserved.

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

Your comments need work.

Fragments do not work well for you. Try writing sentences.

Try to write comments for someone who is very smart, very busy, but isn't
familiar with this area of the codebase. Try to predict what they need to know
and give them some clues about how each part relates to the whole.

Be brief, not cryptic.

[Nils Barth (inactive), 2013/06/26 04:43:37]

Will do.

+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Blink IDL Intermediate Representation (IR) classes

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

End the first line of a docstring comment with a period (or other appropriate
terminal punctuation.)

+
+Also JSON export.
+"""
+
+# Disable attribute hiding check (else JSONEncoder default raises an error)
+# pylint: disable=E0202
+
+import abc
+import json
+import os.path
+import re
+
+
+# Base classes
+
+
+class BaseIdl():
+    """Abstract base class, used for JSON serialization."""
+    __metaclass__ = abc.ABCMeta
+
+    @abc.abstractmethod
+    def to_json(self):

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

Is there a better name for this? It doesn't return JSON per se.

[Nils Barth (inactive), 2013/06/26 04:43:37]

"json_serializable" is more precise; will change.

+        """Return a serializable form of the object.

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

Return -> Returns

+
+        Compatible with Perl IR and JSON import.

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

Wrap text; separate paragraphs with blank lines. You have this random line break
at the end of this sentence.

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

This documentation needs to be improved.

"Compatible with Perl IR and JSON import." isn't really a sentence, it is a
fragment. It is confusing. Just say what you mean in plain language.

You might consider including a path to the definition on the Perl side so people
have a clue to what they need to keep in sync if they're changing this.

Don't make me guess what "JSON import" is. You seem to be saying this method
produces JSON, which is compatible with... JSON. The first rule of tautology
club is the first rule of tautology club.

+        In practice a dictionary, whose keys specify the class."""
+        pass

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

It would be better to raise an exception here. If execution reaches here, it is
a bug in the derived class, right?

[Nils Barth (inactive), 2013/06/26 04:43:37]

The ABC (Abstract Base Class) library handles this, and even better.
The decorator makes this an abstract method, so an exception is raised at object
instantiation if it is unimplemented, rather than at method invocation.

+
+
+class TypedIdlObject(BaseIdl):

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

If this class handles typedefs, maybe it should have typedef in the name.

+    """Auxiliary class for handling typedefs."""

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

Auxiliary is a meaningless word without more context.

+    data_type = None
+    extended_attributes = {}

[dominicc (has gone to gerrit), 2013/06/26 04:20:53]

Won't this hash object be shared by all instances of this class?

[Nils Barth (inactive), 2013/06/26 04:43:37]

(Just added this to silence lint errors; probably better to just disable the
warnings.)
This should work, as I assign to instance attributes, so the class attributes
never get used.

These attributes actually are supposed to be abstract attributes, though I'm not
sure the syntax for them (they can be omitted from the declaration; this is just
for clarity).

+
+    def apply_typedefs(self, typedefs):
+        """Applies typedefs to object itself (e.g., return type of function).
+
+        Override if calling on parameters as well."""
+        new_extended_attributes = {}
+        # Convert string representation to and from a Type object
+        # to handle parsing
+        data_type_object = Type.from_string(self.data_type)
+        base_type = data_type_object.base_type
+        if base_type in typedefs:
+            replacement_type = typedefs[base_type]
+            data_type_object.base_type = replacement_type.data_type
+            new_extended_attributes = replacement_type.extended_attributes
+        self.data_type = str(data_type_object)
+        self.extended_attributes.update(new_extended_attributes)
+        if new_extended_attributes:
+            raise ValueError('Extended attributes in a typedef are untested!')
+
+
+# IDL classes
+
+
+class IdlDocument(BaseIdl):
+    def __init__(self, callback_functions=None, enumerations=None, file_name=None, interfaces=None, typedefs=None):
+        self.callback_functions = callback_functions or []
+        self.enumerations = enumerations or []
+        if file_name:
+            self.file_name = os.path.abspath(file_name)
+        self.interfaces = interfaces or []
+        if typedefs:
+            self.apply_typedefs(typedefs)
+
+    def apply_typedefs(self, typedefs):
+        for callback_function in self.callback_functions:
+            callback_function.apply_typedefs(typedefs)
+        for interface in self.interfaces:
+            interface.apply_typedefs(typedefs)
+
+    def to_json(self):
+        return {
+                'idlDocument::callbackFunctions': self.callback_functions,
+                'idlDocument::enumerations': self.enumerations,
+                'idlDocument::fileName': self.file_name,
+                'idlDocument::interfaces': self.interfaces,
+                }
+
+
+class CallbackFunction(TypedIdlObject):
+    def __init__(self, name=None, data_type=None, parameters=None):
+        """parameters: List of DomParameters"""
+        self.data_type = data_type
+        self.name = name or ""  # FIXME: is "" needed?
+        self.parameters = parameters or []
+
+    def apply_typedefs(self, typedefs):
+        TypedIdlObject.apply_typedefs(self, typedefs)
+        for parameter in self.parameters:
+            parameter.apply_typedefs(typedefs)
+        raise ValueError('Typedefs in CallbackFunctions are untested!')
+
+    def to_json(self):
+        return {
+            'callbackFunction::name': self.name,
+            'callbackFunction::type': self.data_type,
+            'callbackFunction::parameters': self.parameters,
+            }
+
+
+class DomEnum(BaseIdl):
+    def __init__(self, name=None, values=None):
+        """name: enumeration identifier
+        values: enumeration values, list of unique strings
+        """
+        self.name = name
+        self.values = values or []
+
+    def to_json(self):
+        return {
+            'domEnum::name': self.name,
+            'domEnum::values': self.values,
+            }
+
+
+class DomInterface(BaseIdl):
+    def __init__(self, name=None, parents=None, constants=None, functions=None, attributes=None, extended_attributes=None, constructors=None, custom_constructors=None, is_exception=None, is_callback=None, is_partial=None):
+        """
+        attributes: list of DomAttributes
+        constants: list of DomConstants
+        constructors: list of DomFunctions
+        custom_constructors: list of DomFunctions
+        functions: list of DomFunctions
+        is_exception: used for exceptions
+        parents: list of strings
+        """
+        self.attributes = attributes or []
+        self.constants = constants or []
+        self.constructors = constructors or []
+        self.custom_constructors = custom_constructors or []
+        self.extended_attributes = extended_attributes or {}
+        self.functions = functions or []
+        self.is_exception = is_exception
+        self.is_callback = is_callback
+        self.is_partial = is_partial
+        self.name = name
+        self.parents = parents or []
+
+    def apply_typedefs(self, typedefs):
+        for constant in self.constants:
+            constant.apply_typedefs(typedefs)
+        for attribute in self.attributes:
+            attribute.apply_typedefs(typedefs)
+        for function in self.functions:
+            function.apply_typedefs(typedefs)
+        for constructor in self.constructors:
+            constructor.apply_typedefs(typedefs)
+        for custom_constructor in self.custom_constructors:
+            custom_constructor.apply_typedefs(typedefs)
+
+    def to_json(self):
+        return {
+            'domInterface::name': self.name,
+            'domInterface::parents': self.parents,
+            'domInterface::constants': self.constants,
+            'domInterface::functions': self.functions,
+            'domInterface::attributes': self.attributes,
+            'domInterface::extendedAttributes': none_to_value_is_missing(self.extended_attributes),
+            'domInterface::constructors': self.constructors,
+            'domInterface::customConstructors': self.custom_constructors,
+            'domInterface::isException': boolean_to_perl(self.is_exception),
+            'domInterface::isCallback': boolean_to_perl(self.is_callback),
+            'domInterface::isPartial': self.is_partial,
+            }
+
+
+class DomAttribute(TypedIdlObject):
+    def __init__(self, data_type=None, name=None, is_nullable=None, is_static=None, is_read_only=None, getter_exceptions=None, setter_exceptions=None, extended_attributes=None):
+        """data_type: Attribute type (including namespace), string or UnionType
+        is_nullable: (T?)
+        getter_exceptions: Possibly raised exceptions
+        setter_exceptions: Possibly raised exceptions
+        """
+        self.data_type = data_type
+        self.extended_attributes = extended_attributes or {}
+        self.getter_exceptions = getter_exceptions or []
+        self.is_nullable = is_nullable
+        self.is_static = is_static
+        self.is_read_only = is_read_only
+        self.name = name
+        self.setter_exceptions = setter_exceptions or []
+
+    def to_json(self):
+        return {
+            'domAttribute::extendedAttributes': none_to_value_is_missing(self.extended_attributes),
+            'domAttribute::getterExceptions': self.getter_exceptions,
+            'domAttribute::isNullable': self.is_nullable,
+            'domAttribute::isReadOnly': self.is_read_only,
+            'domAttribute::isStatic': self.is_static,
+            'domAttribute::name': self.name,
+            'domAttribute::setterExceptions': self.setter_exceptions,
+            'domAttribute::type': self.data_type,
+            }
+
+
+class DomConstant(TypedIdlObject):
+    def __init__(self, name=None, data_type=None, value=None, extended_attributes=None):
+        self.data_type = data_type
+        self.extended_attributes = extended_attributes or {}
+        self.name = name
+        self.value = value
+
+    def to_json(self):
+        return {
+            'domConstant::extendedAttributes': none_to_value_is_missing(self.extended_attributes),
+            'domConstant::name': self.name,
+            'domConstant::type': self.data_type,
+            'domConstant::value': self.value,
+            }
+
+
+class DomFunction(TypedIdlObject):
+    def __init__(self, is_static=None, name=None, data_type=None, extended_attributes=None, specials=None, parameters=None, overloaded_index=None):
+        """parameters: List of DomParameters"""
+        self.is_static = is_static
+        self.name = name or ""
+        self.data_type = data_type
+        self.extended_attributes = extended_attributes or {}
+        self.specials = specials or []
+        self.parameters = parameters or []
+        self.overloaded_index = overloaded_index
+
+    def apply_typedefs(self, typedefs):
+        TypedIdlObject.apply_typedefs(self, typedefs)
+        for parameter in self.parameters:
+            parameter.apply_typedefs(typedefs)
+
+    def to_json(self):
+        return {
+            'domFunction::extendedAttributes': none_to_value_is_missing(self.extended_attributes),
+            'domFunction::isStatic': boolean_to_perl(self.is_static),
+            'domFunction::name': self.name,
+            'domFunction::overloadedIndex': self.overloaded_index,
+            'domFunction::parameters': self.parameters,
+            'domFunction::specials': self.specials,
+            'domFunction::type': self.data_type,
+            }
+
+
+class DomParameter(TypedIdlObject):
+    def __init__(self, name=None, data_type=None, extended_attributes=None, is_optional=False, is_nullable=None, is_variadic=False):
+        """Used to represent a map of 'variable name' <-> 'variable type'
+
+        data_type: string or UnionType
+        is_optional: (optional T)
+        is_nullable: (T?)
+        is_variadic: (long... numbers)
+        """
+        self.data_type = data_type
+        self.extended_attributes = extended_attributes or {}
+        # FIXME: boolean values are inconsistent (due to Perl),
+        # being sometimes True, False, or None (Perl: 1, 0, undef)
+        # Should all default to False, and be either True or False
+        if is_optional is None:
+            is_optional = False
+            if is_variadic is None:
+                is_variadic = False
+        self.is_nullable = is_nullable
+        # FIXME: can these be in to_json instead?
+        self.is_optional = boolean_to_perl(is_optional)
+        self.is_variadic = boolean_to_perl(is_variadic)
+        self.name = name
+
+    def to_json(self):
+        return {
+            'domParameter::extendedAttributes': none_to_value_is_missing(self.extended_attributes),
+            'domParameter::isNullable': self.is_nullable,
+            'domParameter::isOptional': self.is_optional,
+            'domParameter::isVariadic': self.is_variadic,
+            'domParameter::name': self.name,
+            'domParameter::type': self.data_type,
+            }
+
+# Type classes
+
+
+class Type():
+    # FIXME: replace Type strings with these objects,
+    # so don't need to parse everywhere types are used.
+    # Types are stored internally as strings, not objects,
+    # e.g., as 'sequence<Foo>' or 'Foo[]',
+    # hence need to parse the string whenever a type is used.
+    # FIXME: incorporate Nullable, Variadic, etc.
+    # FIXME: properly should nest types
+    # Formally types are nested, e.g., short?[] vs. short[]?,
+    # but in practice these complex types aren't used and can treat
+    # as orthogonal properties.
+    def __init__(self, base_type=None, is_array=False, is_sequence=False):
+        if is_array and is_sequence:
+            raise ValueError('Array of Sequences not allowed.')
+        self.base_type = base_type
+        self.is_array = is_array
+        self.is_sequence = is_sequence
+
+    def __str__(self):
+        type_string = self.base_type
+        if self.is_array:
+            type_string += '[]'
+        if self.is_sequence:
+            type_string = 'sequence<%s>' % type_string
+        return type_string
+
+    @classmethod
+    def from_string(cls, type_string):
+        is_array = False
+        if type_string.endswith('[]'):
+            is_array = True
+            type_string = type_string[:-2]
+
+        is_sequence = False
+        sequence_re = r'^sequence<([^>]*)>$'
+        sequence_match = re.match(sequence_re, type_string)
+        if sequence_match:
+            is_sequence = True
+            type_string = sequence_match.group(1)
+        base_type = type_string
+        return cls(base_type=base_type, is_array=is_array, is_sequence=is_sequence)
+
+
+class Typedef():
+    # Not exposed in bindings, internal to IDL parsing
+    def __init__(self, extended_attributes=None, data_type=None):
+        self.extended_attributes = extended_attributes or {}
+        self.data_type = data_type
+
+
+class UnionType(BaseIdl):
+    def __init__(self, union_member_types=None):
+        """union_member_types: list of string or UnionType"""
+        self.union_member_types = union_member_types or []
+
+    def to_json(self):
+        return {
+            'UnionType::unionMemberTypes': self.union_member_types,
+            }
+
+
+# Perl JSON compatiblity functions
+# FIXME: remove when Perl removed
+
+def none_to_value_is_missing(extended_attributes):
+    # Perl IDL Parser uses 'VALUE_IS_MISSING' for null values in
+    # extended attributes, so add this as a filter when exporting to JSON.
+    new_extended_attributes = {}
+    for key, value in extended_attributes.iteritems():
+        if value is None:
+            new_extended_attributes[key] = 'VALUE_IS_MISSING'
+        else:
+            new_extended_attributes[key] = value
+    return new_extended_attributes
+
+
+def boolean_to_perl(value):
+    # Perl stores booleans as 1, 0, or undefined (JSON null);
+    # convert to this format
+    if value is None:
+        return None
+    return int(value)
+
+# JSON export
+# FIXME: remove when Perl removed
+# (so no longer using JSON as intermediate format)
+
+
+class IdlEncoder(json.JSONEncoder):
+    def default(self, obj):
+        if isinstance(obj, BaseIdl):
+            return obj.to_json()
+        return json.JSONEncoder.default(self, obj)
+
+
+def ir_to_json(ir, debug=False):
+    if debug:
+        # More legible
+        return json.dumps(ir, cls=IdlEncoder, sort_keys=True, indent=4)
+    return json.dumps(ir, cls=IdlEncoder, sort_keys=True, separators=(',', ':'))