Create a TokenStreamRewriter class.

Create a TokenStreamRewriter class.

This class will allow parser error recovery to insert synthetic tokens
into the token stream.  It makes use of Token.previousToken when it's
available, but does not depend on it; when it's not available, it
falls back on heuristics which should still be fast, and which don't
impose any performance burden until the first synthetic token is
inserted.

R=danrubel@google.com

Committed: https://github.com/dart-lang/sdk/commit/b505603f5f38f28c22effb1c0644731b4804311c

[Paul Berry, 2017-03-08 17:37:27]

paulberry@google.com changed reviewers:
+ ahe@google.com, danrubel@google.com

[danrubel, 2017-03-08 21:32:11]

LGTM

[Paul Berry, 2017-03-08 21:40:35]

Description was changed from

==========
Create a TokenStreamRewriter class.

This class will allow parser error recovery to insert synthetic tokens
into the token stream.  It makes use of Token.previousToken when it's
available, but does not depend on it; when it's not available, it
falls back on heuristics which should still be fast, and which don't
impose any performance burden until the first synthetic token is
inserted.
==========

to

==========
Create a TokenStreamRewriter class.

This class will allow parser error recovery to insert synthetic tokens
into the token stream.  It makes use of Token.previousToken when it's
available, but does not depend on it; when it's not available, it
falls back on heuristics which should still be fast, and which don't
impose any performance burden until the first synthetic token is
inserted.

R=danrubel@google.com

Committed:
https://github.com/dart-lang/sdk/commit/b505603f5f38f28c22effb1c0644731b4804311c
==========

[Paul Berry, 2017-03-08 21:40:36]

Committed patchset #1 (id:1) manually as
b505603f5f38f28c22effb1c0644731b4804311c (presubmit successful).

[ahe, 2017-03-09 13:47:08]

lgtm

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/lib/src/fasta...
File pkg/front_end/lib/src/fasta/parser/token_stream_rewriter.dart (right):

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/parser/token_stream_rewriter.dart:47: // Note: even
though previousToken is deprecated, we need to hook it up in
A trick I've used in the past:

{
  // Note: even though previousToken is deprecated, we need to hook it up in
  // case any uses of it remain.  Once previousToken is removed it should be
  // safe to remove this block of code.
  insertionPoint.previousToken = newToken;
  newToken.previousToken = previous;
}

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/parser/token_stream_rewriter.dart:84: /// Uses
heuristics to skip matching {}, [], (), and <> if possible.
Wrap syntax in back ticks?

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/test/fasta/pa...
File pkg/front_end/test/fasta/parser/token_stream_rewriter_test.dart (right):

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/test/fasta/pa...
pkg/front_end/test/fasta/parser/token_stream_rewriter_test.dart:9: import
'package:test_reflective_loader/test_reflective_loader.dart';
At some point we should talk about test frameworks.

[Paul Berry, 2017-03-09 16:44:39]

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/lib/src/fasta...
File pkg/front_end/lib/src/fasta/parser/token_stream_rewriter.dart (right):

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/parser/token_stream_rewriter.dart:47: // Note: even
though previousToken is deprecated, we need to hook it up in
On 2017/03/09 13:47:07, ahe wrote:
> A trick I've used in the past:
> 
> {
>   // Note: even though previousToken is deprecated, we need to hook it up in
>   // case any uses of it remain.  Once previousToken is removed it should be
>   // safe to remove this block of code.
>   insertionPoint.previousToken = newToken;
>   newToken.previousToken = previous;
> }

Good idea--that's much clearer.  I'll fix in a follow-up CL.

https://codereview.chromium.org/2732193007/diff/1/pkg/front_end/lib/src/fasta...
pkg/front_end/lib/src/fasta/parser/token_stream_rewriter.dart:84: /// Uses
heuristics to skip matching {}, [], (), and <> if possible.
On 2017/03/09 13:47:08, ahe wrote:
> Wrap syntax in back ticks?

Will address in a follow-up CL.