Code generation for extensions api

tools/json_schema_compiler/code.py

Index: tools/json_schema_compiler/code.py
diff --git a/tools/json_schema_compiler/code.py b/tools/json_schema_compiler/code.py
new file mode 100644
index 0000000000000000000000000000000000000000..d9c48e7316a77d681cfa18313751a05fd7964181
--- /dev/null
+++ b/tools/json_schema_compiler/code.py
@@ -0,0 +1,232 @@
+# Copyright (c) 2011 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+"""Utilities for code generation.
+
+Main classes are Code for appending code and
+TypeManager for managing types for things in model."""
+
+from model import PropertyType
+from datetime import datetime
+
+
+CHROMIUM_LICENSE = (
+"""// Copyright (c) %d The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.""" % datetime.now().year
+)
+WARNING_MESSAGE = """// GENERATED FROM THE API DEFINITION IN
+//   %s
+// DO NOT EDIT."""
+
+INDENT_SIZE = 2
+COMMENT_LENGTH = 80
+class Code(object):
+  """A convenience object for appending code.
+
+  Logically each object should be a block of code."""
+  def __init__(self):
+    self.code = []
+    # TODO(calamity): Indent stack probably isn't necessary
+    self._indent_list = []
+    self.indent_level = 0
+
+  def append(self, line=''):
+    """Appends a line of code at the current indent level or just a
+    newline if line is not specified.
+
+    This will strip trailing whitespace."""
+    self.code.append(((' ' * self.indent_level) + line).rstrip())
+    return self
+
+  def add(self, obj):
+    """Concatenate another Code object onto this one.

[not at google - send to devlin, 2012/01/13 00:07:27]

call it "concat" then?  add is an odd name.

[calamity, 2012/01/16 04:01:06]

Done.

+
+    Appends the code at the current indent level. Will fail if there are any
+    un-interpolated format specifiers eg %s, %(something)s which helps
+    isolate any strings that haven't been substituted.
+    """
+    if not isinstance(obj, Code):
+      raise TypeError()
+    for line in obj.code:
+      self.code.append(((' ' * self.indent_level) + line % ()).rstrip())
+
+    return self
+
+  # TODO(calamity): is variable indent size necessary/a good idea?

[not at google - send to devlin, 2012/01/12 06:01:05]

yeah, doesn't look like you're using it anyway. might as well delete it and add
it back if you need it.

[calamity, 2012/01/12 22:59:20]

Done.

+  def sblock(self, line, indent=INDENT_SIZE):

[not at google - send to devlin, 2012/01/12 06:01:05]

I don't like the names "sblock" and "eblock" much anyway. How about a slightly
different API for Code in the first place:

Have a block() method, which returns a new, indented Code object that you can
append to. In other words, code like this:

c.append("foo")
c.sblock("bar1")
c.append("bar2")
c.append("bar3")
c.eblock()
c.append("baz")

would become like this:

c.append("foo")
c.block() \
  .append("bar1") \
  .append("bar2") \
  .append("bar3")
c.append("baz")

or

c.append("foo")
indented = c.block()
indented.append("bar1")
indented.append("bar2")
indented.append("bar3")
c.append("bar")

likewise, rather than add()ing a Code to the code object, you'd get a new one,
like "section()" or something -- which I believe you only need to be able to
substitute names across sections of code, right?

You'd then factor the code generation to take the Code to generate into, rather
than creating a new one internally if needed.  I.e. rather than

def generate_some_code(self):
  c = Code()
  ...
  return c

c.add(generate_some_code())

you'd do

def generate_some_code(self, c):
  ...

generate_some_code(c.section())

[calamity, 2012/01/12 22:59:20]

Can you explain what you dislike about the current API?

If it's indenting, I've only recently discovered that I've been a total noob and
you can just go
(c.append("foo")
  .sblock("bar1")
    .append("bar2")
    .append("bar3")
  .eblock()
  .append("baz")
)

The reason I didn't do this initially is that I didn't know that the python
parser would fail without the enclosing parentheses. Pretty lame.


On 2012/01/12 06:01:05, kalman wrote:
I put those names because they worked out to 6 characters which is the same as
append
Considered passing the same code object everywhere, looked messy because
functions have a fair number of params already.

[not at google - send to devlin, 2012/01/13 00:07:27]

Oh cool, I didn't know that either.  I was assuming you had to use \ everywhere.

Anyway -- I just find the use of sblock/eblock a bit confusing, particularly
passing it a line. Do you really need that?  How about something like

(c.append('foo')
  .append('bar1').start_block()
    .append('bar2')
    .append('bar3').end_block()
  .append('baz'))
Cool, you can avoid it if you take my suggestion above.
Heh, well parameter arguments are cheap.  I wouldn't worry about that.

But yeah, I'm just being an API astronaut*; I think that what I wrote up there
is neat, but after sleeping, don't worry about it; it's too much work to go and
rewrite stuff.


* as opposed to an architecture astronaut.

[not at google - send to devlin, 2012/01/13 00:09:42]

Or similarly,

(c.append('foo')
  .append('bar1').indent()
    .append('bar2')
    .append('bar3').unindent()
  .append('baz'))

hah.  I prefer the aesthetic of the former (start_block/end_block) but
indent/unindent might be more accurate.  The "hah" is because you had this a
while ago.

[calamity, 2012/01/13 00:40:23]

It was more error proof. Calling unindent _after_ the append is problems.

Especially since the reversal is necessary. You wouldn't actually call
.append.end_block(), you'd call .end_block().append() when there are conditions
so the end_block is always run.

 (c.append('foo')
   .append('bar1').start_block()
 )
   if condition1:
     (c.append('bar2')
       .append('bar3')
     )
   else:
     (c.append('bar4')
       .append('bar5')
     )      
 c.end_block().append('baz'))
   ^^^^^^^^^^^^^^^^^^
       Fail bait.
I'm also more partial to a merge-y approach because each function is more
encapsulated and it becomes very obvious where things mess up. Otherwise for the
same safety I'd need checks to enforce code segments not touching each other
inside the Code class.

[not at google - send to devlin, 2012/01/13 02:14:09]

I don't understand what you mean... surely it's pretty clear what (un)indent
would do?

Never mind, though.
Fair enough.

+    """Starts a code block.
+
+    Adds a line of code and then increases the indent level."""
+    self.append(line)
+    self._indent_list.append(self.indent_level)
+    self.indent_level += indent
+    return self
+
+  def eblock(self, line=''):
+    """Ends a code block.
+
+    Decreases the indent level and then adds the line of code.
+    """
+    # TODO(calamity): Decide if type checking is necessary
+    #if not isinstance(line, basestring):
+    #  raise TypeError
+    self.indent_level = self._indent_list.pop()
+
+    self.append(line)
+    return self
+
+  def comment(self, comment):
+    """Adds the given string as a comment.
+
+    Will split the comment if it's too long. Use mainly for variable length
+    comments. Otherwise just use code.append('// ...') for comments.
+    """
+    comment_symbol = '// '
+    max_len = COMMENT_LENGTH - self.indent_level - len(comment_symbol)
+    while len(comment) >= max_len:
+      line = comment[0:max_len]
+      last_space = line.rfind(' ')
+      if last_space != -1:
+        line = line[0:last_space]
+        comment = comment[last_space + 1:]
+      else:
+        comment = comment[max_len:]
+      self.append(comment_symbol + line)
+    self.append(comment_symbol + comment)

[not at google - send to devlin, 2012/01/12 06:01:05]

return self

[calamity, 2012/01/12 22:59:20]

Done.

+
+  def substitute(self, d):
+    """Goes through each line and interpolates using the given dict.

[not at google - send to devlin, 2012/01/13 00:07:27]

Explain why you need to have this method at all, rather than just using normal
string interpolation everywhere.

[calamity, 2012/01/16 04:01:06]

Done.

+
+    Raises type error if passed something that isn't a dict."""
+    if not isinstance(d, dict):
+      raise TypeError('Passed argument is not a dictionary: ' + d)
+    for i, line in enumerate(self.code):
+      # Only need to check %s because arg is a dict and python will allow
+      # '%s %(named)s' but just about nothing else
+      if '%s' in self.code[i] or '%r' in self.code[i]:
+        raise TypeError('%s or %r found in substitution.'
+            'Named arguments only. Use %%s to escape')
+      self.code[i] = line % d

[not at google - send to devlin, 2012/01/12 06:01:05]

return self

[calamity, 2012/01/12 22:59:20]

Done.

+
+  def to_string(self):

[not at google - send to devlin, 2012/01/12 06:01:05]

weird to have this method when there's the python __str__ for this idiom.

call it "render" or something.

[calamity, 2012/01/12 22:59:20]

Done.

+    """Returns the code joined together as a string."""
+    return '\n'.join(self.code)
+
+class TypeManager(object):

[not at google - send to devlin, 2012/01/12 06:01:05]

this should be in its own file.

so should most classes, really.  use your judgement.

[calamity, 2012/01/12 22:59:20]

Done.

+  """Manages the types of properties and provides utlities for getting the
+  C++ type out of a model.PropertyT."""
+  def __init__(self, namespace, model):
+    self.model = model
+    self.types = model.types
+    self.namespace = namespace
+
+  # TODO(calamity): Handle ANY
+  def get_type(self, prop):
+    """Translates a json_type into its C++ equivalent.
+
+    If REF types from different namespaces are referenced, will resolve
+    using self.types.
+    """
+    simple_c_types = {
+      'boolean': 'bool',
+      'integer': 'int',
+      'double': 'double',
+      'string': 'std::string',
+    }
+    if prop.type == PropertyType.REF:
+      ref_type = self.types.get(prop.json_type)
+      if not ref_type:
+        raise KeyError('Cannot find referenced type: %s' % prop.json_type)
+      if self.namespace != ref_type:
+        return '%s::%s' % (ref_type.filename, prop.json_type)
+      else:
+        return '%s' % prop.json_type
+    elif prop.type == PropertyType.FUNDAMENTAL:
+      return simple_c_types[prop.json_type]
+    elif prop.type == PropertyType.ARRAY:
+      return 'std::vector<%s>' % self.get_type(prop.item_type)
+    elif prop.type == PropertyType.OBJECT:
+      return cpp_name(prop.name)
+    # TODO(calamity): choices
+    else:
+      raise NotImplementedError
+
+  def parameter_declaration(self, param, type_modifiers=None,
+      default_format='%(type)s %(name)s'):
+    """Returns a string that declares a parameter. Used where the parameter type
+    isn't clear and covering all cases would be ugly.
+
+    Can be given a dictionary with PropertyType keys and format string values
+    where %(type)s and %(name)s denote the C++ type and parameter
+    name respectively.
+
+    ParamFormat has some common formats.
+    """
+    if not type_modifiers:
+      type_modifiers = {}
+    return (type_modifiers.get(param.type, default_format)) % {
+        'type': self.get_type(param), 'name': param.name}
+
+  def get_generic_type(self, prop):
+    """Returns the c_type of an object suitable for putting in <%s>.
+
+    Will add a space to inner generics to prevent operator ambiguity.
+    """
+    prop_type = self.get_type(prop)
+    if prop_type[-1] == '>':
+      return '%s ' % prop_type
+    else:
+      return '%s' % prop_type
+
+  def resolve_generated_includes(self):
+    """Returns the #include lines for self.namespace using the other
+    namespaces in self.model.
+    """
+    dependencies = set()
+    for function in self.namespace.functions.values():
+      for param in function.params:
+        dependencies |= self.type_dependencies(param)
+      dependencies |= self.type_dependencies(function.callback.param)
+    for tipe in self.namespace.types.values():
+      for prop in tipe.properties.values():
+        dependencies |= self.type_dependencies(prop)
+
+    includes = Code()
+    for dependency in dependencies:
+      dependency_namespace =  self.types[dependency]
+      if dependency_namespace != self.namespace:
+        includes.append('#include "%s/%s.h"' % (
+            dependency_namespace.parent_dir,
+            dependency_namespace.filename))
+    return includes
+
+  def type_dependencies(self, prop):
+    """Gets all the type dependencies of a single property.
+
+    Will also get type dependencies from subproperties"""
+    deps = set()
+    if prop:
+      if prop.type == PropertyType.REF:
+        deps.add(prop.json_type)
+      elif prop.type == PropertyType.ARRAY:
+        deps = self.type_dependencies(prop.item_type)
+      elif prop.type == PropertyType.OBJECT:
+        for p in prop.properties.values():
+          deps |= self.type_dependencies(p)
+    return deps
+
+def cpp_name(s):
+  """Translates a namespace name or function name into something more
+  suited to C++.
+
+  eg experimental.downloads -> Experimental_Downloads
+  updateAll -> UpdateAll.
+  """
+  return '_'.join([x[0].upper() + x[1:] for x in s.split('.')])
+
+class ParamFormat(object):

[not at google - send to devlin, 2012/01/12 06:01:05]

ditto

[calamity, 2012/01/16 04:01:06]

Done.

+  """Common parameter modifier formats to be used with
+  TypeManager.parameter_declaration"""
+  POINTER = '%(type)s* %(name)s'
+  REFERENCE = '%(type)s& %(name)s'