fasta scanner recovery and error code translation improvements

fasta scanner recovery and error code translation improvements

* cleanup defaultRecoveryStrategy next/previous and eof

* add SymbolToken.eof() constructor to ensure that EOF tokens
  are correctly self referencing

* add fasta.scanString method to match scan method

* add fasta.ScannerResult.hasErrors field
  so that scan and scanString callers can quickly check for errors

* new scanner replacement test
  which scans using fasta then asserts as if it were from analyzer

* improve SymbolToken.toString() for EOF
* update analyzer test :: fasta scanner considers "<" to be an opener
* extract and move translateErrorToken into front_end
* address comments in https://codereview.chromium.org/2763833002

R=ahe@google.com, paulberry@google.com

Committed: https://github.com/dart-lang/sdk/commit/186c6f23bd2987b3a356946fa6706f38a52d7e40

[danrubel, 2017-03-22 17:09:46]

danrubel@google.com changed reviewers:
+ ahe@google.com, paulberry@google.com

[ahe, 2017-03-23 11:52:37]

lgtm

Only nits below.

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/f...
File pkg/front_end/lib/src/fasta/scanner.dart (right):

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/f...
pkg/front_end/lib/src/fasta/scanner.dart:82: // does the Recover API need to be
adjusted for a String source?
Since we only need the bytes when performing error recovery, we could pass in
the string source here and convert it to bytes if calling recover.

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/s...
File pkg/front_end/lib/src/scanner/errors.dart (right):

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/s...
pkg/front_end/lib/src/scanner/errors.dart:64: /**
Consistency: use either /// or /**

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/s...
pkg/front_end/lib/src/scanner/errors.dart:67: * The [arguments] are any
arguments needed to complete the error message
Add a period to the end of this sentence.

[Paul Berry, 2017-03-23 16:20:23]

lgtm

[danrubel, 2017-03-24 15:40:29]

Description was changed from

==========
fasta scanner recovery and error code translation improvements

* cleanup defaultRecoveryStrategy next/previous and eof

* add SymbolToken.eof() constructor to ensure that EOF tokens
  are correctly self referencing

* add fasta.scanString method to match scan method

* add fasta.ScannerResult.hasErrors field
  so that scan and scanString callers can quickly check for errors

* new scanner replacement test
  which scans using fasta then asserts as if it were from analyzer

* improve SymbolToken.toString() for EOF
* update analyzer test :: fasta scanner considers "<" to be an opener
* extract and move translateErrorToken into front_end
* address comments in https://codereview.chromium.org/2763833002
==========

to

==========
fasta scanner recovery and error code translation improvements

* cleanup defaultRecoveryStrategy next/previous and eof

* add SymbolToken.eof() constructor to ensure that EOF tokens
  are correctly self referencing

* add fasta.scanString method to match scan method

* add fasta.ScannerResult.hasErrors field
  so that scan and scanString callers can quickly check for errors

* new scanner replacement test
  which scans using fasta then asserts as if it were from analyzer

* improve SymbolToken.toString() for EOF
* update analyzer test :: fasta scanner considers "<" to be an opener
* extract and move translateErrorToken into front_end
* address comments in https://codereview.chromium.org/2763833002

R=ahe@google.com, paulberry@google.com

Committed:
https://github.com/dart-lang/sdk/commit/186c6f23bd2987b3a356946fa6706f38a52d7e40
==========

[danrubel, 2017-03-24 15:40:30]

Committed patchset #3 (id:40001) manually as
186c6f23bd2987b3a356946fa6706f38a52d7e40 (presubmit successful).

[danrubel, 2017-03-24 15:44:20]

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/f...
File pkg/front_end/lib/src/fasta/scanner.dart (right):

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/f...
pkg/front_end/lib/src/fasta/scanner.dart:82: // does the Recover API need to be
adjusted for a String source?
On 2017/03/23 11:52:36, ahe wrote:
> Since we only need the bytes when performing error recovery, we could pass in
> the string source here and convert it to bytes if calling recover.

Done.

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/s...
File pkg/front_end/lib/src/scanner/errors.dart (right):

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/s...
pkg/front_end/lib/src/scanner/errors.dart:64: /**
On 2017/03/23 11:52:36, ahe wrote:
> Consistency: use either /// or /**

Good point. Changed /// to /** below to match the rest of the file.

https://codereview.chromium.org/2767083002/diff/20001/pkg/front_end/lib/src/s...
pkg/front_end/lib/src/scanner/errors.dart:67: * The [arguments] are any
arguments needed to complete the error message
On 2017/03/23 11:52:36, ahe wrote:
> Add a period to the end of this sentence.

Done.