Add the ability to convert an analyzer token stream to a Fasta one.

Add the ability to convert an analyzer token stream to a Fasta one.

This will be needed in order to integrate Fasta into the existing
analyzer codebase, because the analyzer task model considers the
parser and scanner to be separate black boxes with an analyzer token
stream in between.  In order for Fasta to be able to drop into those
black boxes, we will need to be able to convert Fasta tokens to
analyzer tokens after scanning, and then back to Fasta tokens prior to
parsing.

Tested via a "round trip" test that converts the analyzer token stream
into a Fasta token stream and then back.

Note that the transformation is not perfect; it currently fails to
recreate error tokens and it fails to hook up the connection between a
BeginGroup token and the corresponding end token.  But it is enough to
be able to pass tests for now.  It will be improved (and tests will be
expanded) in future CLs.

R=ahe@google.com

Committed: https://github.com/dart-lang/sdk/commit/f0e657015dd72ec0ea97080b8490cda075053743

[Paul Berry, 2017-02-11 13:43:28]

paulberry@google.com changed reviewers:
+ ahe@google.com

[ahe, 2017-02-12 10:07:35]

lgtm

https://codereview.chromium.org/2689733003/diff/1/pkg/front_end/lib/src/fasta...
File pkg/front_end/lib/src/fasta/analyzer/token_utils.dart (right):

https://codereview.chromium.org/2689733003/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/analyzer/token_utils.dart:159: return
internalError('Unrecognized keyword: ${token.lexeme}');
For consistency: "Unrecognized keyword: '${token.lexeme}'."

[Paul Berry, 2017-02-13 17:44:33]

https://codereview.chromium.org/2689733003/diff/1/pkg/front_end/lib/src/fasta...
File pkg/front_end/lib/src/fasta/analyzer/token_utils.dart (right):

https://codereview.chromium.org/2689733003/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/analyzer/token_utils.dart:159: return
internalError('Unrecognized keyword: ${token.lexeme}');
On 2017/02/12 10:07:35, ahe wrote:
> For consistency: "Unrecognized keyword: '${token.lexeme}'."

Done.

[Paul Berry, 2017-02-13 18:24:53]

Description was changed from

==========
Add the ability to convert an analyzer token stream to a Fasta one.

This will be needed in order to integrate Fasta into the existing
analyzer codebase, because the analyzer task model considers the
parser and scanner to be separate black boxes with an analyzer token
stream in between.  In order for Fasta to be able to drop into those
black boxes, we will need to be able to convert Fasta tokens to
analyzer tokens after scanning, and then back to Fasta tokens prior to
parsing.

Tested via a "round trip" test that converts the analyzer token stream
into a Fasta token stream and then back.

Note that the transformation is not perfect; it currently fails to
recreate error tokens and it fails to hook up the connection between a
BeginGroup token and the corresponding end token.  But it is enough to
be able to pass tests for now.  It will be improved (and tests will be
expanded) in future CLs.
==========

to

==========
Add the ability to convert an analyzer token stream to a Fasta one.

This will be needed in order to integrate Fasta into the existing
analyzer codebase, because the analyzer task model considers the
parser and scanner to be separate black boxes with an analyzer token
stream in between.  In order for Fasta to be able to drop into those
black boxes, we will need to be able to convert Fasta tokens to
analyzer tokens after scanning, and then back to Fasta tokens prior to
parsing.

Tested via a "round trip" test that converts the analyzer token stream
into a Fasta token stream and then back.

Note that the transformation is not perfect; it currently fails to
recreate error tokens and it fails to hook up the connection between a
BeginGroup token and the corresponding end token.  But it is enough to
be able to pass tests for now.  It will be improved (and tests will be
expanded) in future CLs.

R=ahe@google.com

Committed:
https://github.com/dart-lang/sdk/commit/f0e657015dd72ec0ea97080b8490cda075053743
==========

[Paul Berry, 2017-02-13 18:24:54]

Committed patchset #1 (id:1) manually as
f0e657015dd72ec0ea97080b8490cda075053743 (presubmit successful).