Creating an ANTLR grammar for Dart, potentially the syntax spec.

Creating an ANTLR grammar for Dart, potentially the syntax spec.

[eernst, 2017-02-17 13:26:42]

eernst@google.com changed reviewers:
+ floitsch@google.com, lrn@chromium.org

[eernst, 2017-02-17 13:26:43]

I've created an ANTLR grammar for Dart (Dart.g below). It covers the language
except very few things (multi-line comments, multi-line strings, and string
interpolation with braces, `${..}`). Those things are a bit tricky, but I'm sure
we can fix them.

I'd like to propose that we make this grammar the syntactic specification of the
language:

  - It is available as a complete grammar (no need to extract bits and pieces
from the spec).
  - It is machine checkable, and provides diagnostics which are principled and
independent of diverse Dart specific tools. For instance, we will be able to
know immediately whether the grammar is ambiguous.
  - It can be verified using our test suite (it shouldn't be hard to create a
parser using this grammar, and then simply run that on our test files, just like
all the other tools).
  - It allows us to characterize the parsing techniques in a manner that fits
current approaches.

The last part is important because it allows us to specify how parsing is
_supposed_ to work, and it helps us justifying that it is possible to do it in
the specified manner. The tools (say, dart2js/fasta or the analyzer) may use
different techniques, but we then have one source of truth on parsing.
Crucially, ANTLR allows us to make it compatible with the existing
implementation techniques. Just one example: we can use syntactic predicates to
do exactly the same thing as the dart2js/fasta parser does in
`isValidTypeReference` and similar functions.

This would be helpful when fixing the grammar. We know that there are some
glitches, and it is not easy to inspect the grammar as a whole today. Fixes
applied to Dart.g could be checked rigorously (by ANTLR when generating the
lexer/parser, and then relative to our test suite), and we could then go back
and update the grammar snippets in the language specification.

It would also be helpful during language design experiments: We may need to make
several speculative changes to the grammar as we consider language extensions or
modifications, and it would be nice to be able to trust a grammar and try out
the consequences directly.

A note about ANTLR versions: Dart.g gets processed by ANTLR v3 (the tool is
`antlr3`). Unfortunately, we do not seem to have any version of ANTLR v4
available; we probably don't want to put a lot of effort into getting v4
accepted and maintained (and v3 works fine, too, although a lot of smarts are
missing). We also need to use Java 1.7, because Java 1.8-and-up and antlr3 do
not work together.

[eernst, 2017-08-25 15:43:53]

PTAL.

The spec grammar Dart.g has been completed to such a degree that it parses
"almost all" test programs. A file *_parser.status in some directories indicate
which files are not yet covered (in particular, test.py doesn't know how to run
the spec parser, so the spec parser can't parse files that have syntax errors
because they are multi-tests).

I'd like to land this soon, such that (1) it will be possible to refer to the
spec grammar to resolve questions about the grammar (where the grammar snippets
in dartLangSpec.tex have issues in several respects), and (2) it will be
possible to evolve the spec grammar together with the language, such that we
have a working grammar for each version of the language in the obvious place,
and with the same version management as all other elements of Dart.

There is still more work to do in order to make it possible for buildbots to run
the spec parser on all tests (I guess just one platform would be enough), but we
can do that later.

[floitsch, 2017-08-28 09:18:49]

DBC.

https://codereview.chromium.org/2688903004/diff/320001/docs/language/Dart.g
File docs/language/Dart.g (right):

https://codereview.chromium.org/2688903004/diff/320001/docs/language/Dart.g#n...
docs/language/Dart.g:180: :    COVARIANT? finalConstVarOrType identifier
Why is Covariant allowed at the top-level (through
initializedVariableDeclaration)?
Shouldn't that only be allowed for non-statics?

[Lasse Reichstein Nielsen, 2017-08-28 10:51:30]

lrn@google.com changed reviewers:
+ lrn@google.com

[Lasse Reichstein Nielsen, 2017-08-28 10:51:34]

https://codereview.chromium.org/2688903004/diff/320001/docs/language/Dart.g
File docs/language/Dart.g (right):

https://codereview.chromium.org/2688903004/diff/320001/docs/language/Dart.g#n...
docs/language/Dart.g:180: :    COVARIANT? finalConstVarOrType identifier
I believe we handle that in prose: It's an error if `covariant` occurs on
something that is not the parameter of an instance method or setter, or an
instance field, where it applies to the implicit setter (if any).

We do that for other cases too (I believe initializing formals are handled that
way too).

[eernst, 2017-08-28 11:35:16]

Review response.

https://codereview.chromium.org/2688903004/diff/320001/docs/language/Dart.g
File docs/language/Dart.g (right):

https://codereview.chromium.org/2688903004/diff/320001/docs/language/Dart.g#n...
docs/language/Dart.g:180: :    COVARIANT? finalConstVarOrType identifier
On 2017/08/28 10:51:32, Lasse Reichstein Nielsen wrote:
> I believe we handle that in prose: It's an error if `covariant` occurs on
> something that is not the parameter of an instance method or setter, or an
> instance field, where it applies to the implicit setter (if any).
> 
> We do that for other cases too (I believe initializing formals are handled
that
> way too).

Right, it's quite "imprecise" to allow COVARIANT on declaredIdentifier because
it includes all those cases where it makes no sense. In order to handle it
elsewhere, however, we'd probably need to duplicate a rather large number of
non-terminals: We'd need to duplicate declaredIdentifier <-
simpleFormalParameter <- normalFormalParameterNoMetadata <-
normalFormalParameter <- normalFormalParameters <- formalParameterList in order
to be able to use the-nonterminal-that-allows-COVARIANT in instance methods and
the-one-that-doesn't in static methods etc.

So it's probably not worth the trouble to encode this in the grammar.

[eernst, 2017-08-30 10:57:56]

CC to ahe@.

[Lasse Reichstein Nielsen, 2017-08-30 14:32:01]

lrn@google.com changed reviewers:
- lrn@chromium.org

[Lasse Reichstein Nielsen, 2017-09-20 07:50:59]

lgtm