Document the parser's design.

pkg/front_end/lib/src/fasta/parser/parser.dart

 // Copyright (c) 2012, the Dart project authors.  Please see the AUTHORS file
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
 library fasta.parser.parser;
 
 import '../fasta_codes.dart'
     show
         FastaCode,
         FastaMessage,
@@ skipping 164 matching lines @@
 /// [keyword.dart](../scanner/keyword.dart) and ensure the identifier is added
 /// to the keyword table.
 ///
 /// As a consequence of this, one should not use `==` to compare strings in the
 /// parser. One should favor the methods [optional] and [expected] to recognize
 /// keywords or identifiers. In some cases, it's possible to compare a token's
 /// `stringValue` using [identical], but normally [optional] will suffice.
 ///
 /// Historically, we over-used identical, and when identical is used on other
 /// objects than strings, it can often be replaced by `==`.
+///
+/// ## Flexibility, Extensibility, and Specification
+///
+/// The parser is designed to be flexible and extensible. It's methods are

[scheglov, 2017/04/06 15:35:35]

"Its"

[ahe, 2017/04/06 15:52:30]

Done.

+/// designed to be overridden in subclasses, so it can be extended to handle
+/// unspecified language extension or experiments while everything in this file
+/// attempts to follow the specification (unless when it interferes with error
+/// recovery).
+///
+/// We achieve flexibily, extensible, and specification compliance by following
+/// a few rules-of-thumb:
+///
+/// 1. All methods in the parser should be public.
+///
+/// 2. The methods follow the specified grammar, and do not implement custom
+/// extensions, for example, `native`.
+///
+/// 3. The parser doesn't rewrite the token stream (when dealing with `>>`).
+///
+/// ### Implementing Extensions
+///
+/// For various reasons, some Dart language implementations have used
+/// custom/unspecified extensions to the Dart grammar. Examples of this
+/// includes diet parsing, patch files, `native` keyword, and generic
+/// comments. This class isn't supposed to implement any of these
+/// features. Instead it provides hooks for those extensions to be implemented
+/// in subclasses or listeners. Let's examine how diet parsing and `native`
+/// keyword is currently supported by Fasta.
+///
+/// #### Implementation of `native` Keyword
+///
+/// Both dart2js and the Dart VM have used the `native` keyword to mark methods
+/// that couldn't be implemented in the Dart language and needed to be
+/// implemented in JavaScript or C++, respectively. An example of the syntax
+/// extension used by the Dart VM is:
+///
+///     nativeFunction() native "NativeFunction";
+///
+/// When attempting to parse this function, the parser eventually calls
+/// [parseFunctionBody]. This method will report an unrecoverable error to the
+/// listener with the code [codeExpectedFunctionBody]. The listener can then
+/// look at the error code and the token and use the methods in
+/// [dart_vm_native.dart](dart_vm_native.dart) to parse the native syntax.
+///
+/// #### Implementation of Diet Parsing
+///
+/// We say diet parsing when skipping parts of a file. Both dart2js and the

[Johnni Winther, 2017/04/06 13:36:22]

"We say diet parsing when skipping" -> "We call it 'diet parsing' when the
parser skips"

[ahe, 2017/04/06 15:52:30]

Done.

+/// Dart VM has been relying on this from early on as it allows them to more

[scheglov, 2017/04/06 15:35:35]

"have been"?

[ahe, 2017/04/06 15:52:30]

Done.

+/// quickly compile small programs that use small parts of big libraries. It's
+/// also become an integrated part of how Fasta builds up outlines before
+/// starting to parse method bodies.
+///
+/// When looking through this parser, you'll find a number of unused methods
+/// starting with `skip`. These methods are only used by subclasses, such as
+/// [ClassMemberParser](class_member_parser.dart) and
+/// [TopLevelParser](top_level_parser.dart). These methods violate the
+/// principle above about following the specified grammar, and originally lived
+/// in subclasses. However, we realized that these methods were so widely used
+/// and hard to maintain in subclasses, that it made sense to move them here.
+///
+/// ### Specification and Error Recovery
+///
+/// To improve error recovery, the parser will inform the listener of
+/// recoverable errors and continue to parse.  An example of a recoverable
+/// error is:
+///
+///     Error: Asynchronous for-loop can only be used in 'async' or 'async*' methods.

[Johnni Winther, 2017/04/06 13:36:22]

Long line.

+///     main() { await for (var x in []) {} }
+///              ^^^^^
+///
+/// For unrecoverable errors, the parser will ask the listener for help to
+/// recover from the error. We haven't made much progress on these kinds of
+/// errors, so in most cases, the parser aborts by skipping to the end of file.
+///
+/// Historically, this parser has been rather lax in what it allows, and
+/// deferred the enforcement of some syntactical rules to subsequent phases. It
+/// doesn't matter how we got there, only that we've identified that it's
+/// easier if the parser reports as many errors it can, but informs the
+/// listener if the error is recoverable or not.
+///
+/// Currently, the parser is particularly lax when it comes to the order of
+/// modifiers such as `abstract`, `final`, `static`, etc. Historically, dart2js
+/// would handle such errors in later phases. We hope that these cases will go
+/// away as Fasta matures.
 class Parser {
   final Listener listener;
 
   Uri get uri => listener.uri;
 
   bool mayParseFunctionExpressions = true;
 
   bool parseGenericMethodComments = false;
 
   /// Represents parser state: what asynchronous syntax is allowed in the
@@ skipping 3660 matching lines @@
     previous.setNext(firstToken);
     beforeToken = firstToken;
   }
 }
 
 typedef FastaMessage NoArgument(Uri uri, int charOffset);
 
 typedef FastaMessage TokenArgument(Uri uri, int charOffset, Token token);
 
 typedef FastaMessage StringArgument(Uri uri, int charOffset, String string);