Move token stream translation logic into Fasta.

Move token stream translation logic into Fasta.

This logic was initially only used in scanner_fasta_test, however we
are going to need it for other tests (some of which are outside of the
Fasta package), and on a short term basis we may want to use it to
help integrate pieces of Fasta with pieces of the analyzer codebase.
So it needs to be somewhere that's accessible outside of
pkg/front_end/test.

I performed some trivial improvements during the move (such as using
Token.info.kind in preference to "is" checks, and identifying the end
of the input by looking for a nearby EOF token).

R=sigmund@google.com

Committed: https://github.com/dart-lang/sdk/commit/2e435ec020be9c361e45f92e1c5f57dc4f141809

[Paul Berry, 2017-02-10 21:30:43]

paulberry@google.com changed reviewers:
+ ahe@google.com, sigmund@google.com

[Siggi Cherem (dart-lang), 2017-02-10 21:46:31]

lgtm

[Paul Berry, 2017-02-10 21:48:11]

Description was changed from

==========
Move token stream translation logic into Fasta.

This logic was initially only used in scanner_fasta_test, however we
are going to need it for other tests (some of which are outside of the
Fasta package), and on a short term basis we may want to use it to
help integrate pieces of Fasta with pieces of the analyzer codebase.
So it needs to be somewhere that's accessible outside of
pkg/front_end/test.

I performed some trivial improvements during the move (such as using
Token.info.kind in preference to "is" checks, and identifying the end
of the input by looking for a nearby EOF token).
==========

to

==========
Move token stream translation logic into Fasta.

This logic was initially only used in scanner_fasta_test, however we
are going to need it for other tests (some of which are outside of the
Fasta package), and on a short term basis we may want to use it to
help integrate pieces of Fasta with pieces of the analyzer codebase.
So it needs to be somewhere that's accessible outside of
pkg/front_end/test.

I performed some trivial improvements during the move (such as using
Token.info.kind in preference to "is" checks, and identifying the end
of the input by looking for a nearby EOF token).

R=sigmund@google.com

Committed:
https://github.com/dart-lang/sdk/commit/2e435ec020be9c361e45f92e1c5f57dc4f141809
==========

[Paul Berry, 2017-02-10 21:48:13]

Committed patchset #1 (id:1) manually as
2e435ec020be9c361e45f92e1c5f57dc4f141809 (presubmit successful).

[ahe, 2017-02-10 21:56:26]

lgtm

https://codereview.chromium.org/2686363002/diff/1/pkg/front_end/lib/src/fasta...
File pkg/front_end/lib/src/fasta/analyzer/token_utils.dart (right):

https://codereview.chromium.org/2686363002/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/analyzer/token_utils.dart:53: // operations.
Absolutely, we should do something like precedingComment.

https://codereview.chromium.org/2686363002/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/analyzer/token_utils.dart:113: int endOffset =
token.endOffset ?? charOffset;
That sounds bad. Would you mind filing a bug if you have source text that causes
this?

It could be my recent error recovery that's broken.

[Paul Berry, 2017-02-10 22:55:01]

https://codereview.chromium.org/2686363002/diff/1/pkg/front_end/lib/src/fasta...
File pkg/front_end/lib/src/fasta/analyzer/token_utils.dart (right):

https://codereview.chromium.org/2686363002/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/analyzer/token_utils.dart:113: int endOffset =
token.endOffset ?? charOffset;
On 2017/02/10 21:56:25, ahe wrote:
> That sounds bad. Would you mind filing a bug if you have source text that
causes
> this?
> 
> It could be my recent error recovery that's broken.

Filed https://github.com/dart-lang/sdk/issues/28745.