Port parser and scanner fixes from rasta branch.

Port parser and scanner fixes from rasta branch.

R=johnniwinther@google.com

Committed: https://github.com/dart-lang/sdk/commit/ce742a1697661c55115bdb8009a3aee6febb16cb

[ahe, 2017-01-27 23:46:33]

ahe@google.com changed reviewers:
+ johnniwinther@google.com

[ahe, 2017-01-27 23:46:33]

https://codereview.chromium.org/2664593002/diff/1/pkg/fasta/test/rasta/issue_...
File pkg/fasta/test/rasta/issue_000035a.dart (right):

https://codereview.chromium.org/2664593002/diff/1/pkg/fasta/test/rasta/issue_...
pkg/fasta/test/rasta/issue_000035a.dart:5: class C{
You can't see in my browser, but there's two Unicode line separator characters
(U+2028) before the constructor.

You can generate this file with this Dart program:

main() {
  print("class C{\u2028\u2028C();}");
}

Also, you can't see these error messages:

pkg/fasta/test/rasta/issue_000035a.dart:227: Error: The non-ASCII space
character U+2028 can only be used in strings and comments.
pkg/fasta/test/rasta/issue_000035a.dart:228: Error: The non-ASCII space
character U+2028 can only be used in strings and comments.

Notice the positions aren't line numbers, they're character offsets.

[ahe, 2017-01-28 00:33:34]

https://codereview.chromium.org/2664593002/diff/20001/tests/compiler/dart2js/...
File tests/compiler/dart2js/message_kind_helper.dart (left):

https://codereview.chromium.org/2664593002/diff/20001/tests/compiler/dart2js/...
tests/compiler/dart2js/message_kind_helper.dart:25: // If you add something
here, please file a *new* bug report.
Given that we're moving away from this front-end, I think we don't need to worry
about this anymore.

https://codereview.chromium.org/2664593002/diff/20001/tests/compiler/dart2js/...
File tests/compiler/dart2js/message_kind_helper.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/tests/compiler/dart2js/...
tests/compiler/dart2js/message_kind_helper.dart:44: MessageKind.UNMATCHED_TOKEN,
The message is repeated: first during diet parsing, then when parsing the body
again.

I have fixed this problem before (in rasta), but I don't think it matters now.

[Johnni Winther, 2017-01-30 09:04:39]

lgtm

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
File pkg/dart_parser/lib/src/parser.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
pkg/dart_parser/lib/src/parser.dart:1226: reportUnrecoverableError(beginGroup,
ErrorKind.UnmatchedToken);
Why not use the token returned from [reportUnrecoverableError]? You do that in
parseStringPart.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
pkg/dart_parser/lib/src/parser.dart:1869: if (statementDepth++ > 500) {
When does this occur?

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
pkg/dart_parser/lib/src/parser.dart:2187: reportRecoverableError(
When does this occur?

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
File pkg/dart_scanner/lib/src/error_token.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/error_token.dart:145:
AsciiControlCharacterToken(this.character, int charOffset) : super(charOffset);
Long line.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
File pkg/dart_scanner/lib/src/recover.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/recover.dart:44: Token beforeGoodTail;
Add doc to these variables.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/recover.dart:93: // the first good tooken.
tooken -> token

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
File pkg/dart_scanner/lib/src/utf8_bytes_scanner.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/utf8_bytes_scanner.dart:111: * with the byte
[nextByte]. If [advance] is true the current [byteOffset]
Remove doc about [advance].

https://codereview.chromium.org/2664593002/diff/20001/pkg/fasta/lib/src/sourc...
File pkg/fasta/lib/src/source/source_loader.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/fasta/lib/src/sourc...
pkg/fasta/lib/src/source/source_loader.dart:124: Token tokens = await
tokenize(library, suppressLexicalErrors: true);
Why suppress errors here?

https://codereview.chromium.org/2664593002/diff/20001/tests/compiler/dart2js/...
File tests/compiler/dart2js/message_kind_helper.dart (left):

https://codereview.chromium.org/2664593002/diff/20001/tests/compiler/dart2js/...
tests/compiler/dart2js/message_kind_helper.dart:25: // If you add something
here, please file a *new* bug report.
On 2017/01/28 00:33:33, ahe wrote:
> Given that we're moving away from this front-end, I think we don't need to
worry
> about this anymore.

Acknowledged.

[ahe, 2017-01-30 13:26:23]

Thank you, Johhni!

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
File pkg/dart_parser/lib/src/parser.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
pkg/dart_parser/lib/src/parser.dart:1226: reportUnrecoverableError(beginGroup,
ErrorKind.UnmatchedToken);
On 2017/01/30 09:04:38, Johnni Winther wrote:
> Why not use the token returned from [reportUnrecoverableError]? You do that in
> parseStringPart.

Because I'm not trying to change or improve error recovery in that case, I'm
just try to prevent a crash.

I'm reluctant to change these things as long as it has a chance to break
message_test from dart2js.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
pkg/dart_parser/lib/src/parser.dart:1869: if (statementDepth++ > 500) {
On 2017/01/30 09:04:38, Johnni Winther wrote:
> When does this occur?

Deeply nested if statements. See
tests/language/deep_nesting2_negative_test.dart.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_parser/lib/src...
pkg/dart_parser/lib/src/parser.dart:2187: reportRecoverableError(
On 2017/01/30 09:04:38, Johnni Winther wrote:
> When does this occur?

Deeply nested array literals, for example, see:
tests/language/deep_nesting1_negative_test.dart

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
File pkg/dart_scanner/lib/src/error_token.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/error_token.dart:145:
AsciiControlCharacterToken(this.character, int charOffset) : super(charOffset);
On 2017/01/30 09:04:38, Johnni Winther wrote:
> Long line.

Done.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
File pkg/dart_scanner/lib/src/recover.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/recover.dart:44: Token beforeGoodTail;
On 2017/01/30 09:04:38, Johnni Winther wrote:
> Add doc to these variables.

Done.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/recover.dart:93: // the first good tooken.
On 2017/01/30 09:04:38, Johnni Winther wrote:
> tooken -> token

Done.

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
File pkg/dart_scanner/lib/src/utf8_bytes_scanner.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/dart_scanner/lib/sr...
pkg/dart_scanner/lib/src/utf8_bytes_scanner.dart:111: * with the byte
[nextByte]. If [advance] is true the current [byteOffset]
On 2017/01/30 09:04:38, Johnni Winther wrote:
> Remove doc about [advance].

Done.

https://codereview.chromium.org/2664593002/diff/20001/pkg/fasta/lib/src/sourc...
File pkg/fasta/lib/src/source/source_loader.dart (right):

https://codereview.chromium.org/2664593002/diff/20001/pkg/fasta/lib/src/sourc...
pkg/fasta/lib/src/source/source_loader.dart:124: Token tokens = await
tokenize(library, suppressLexicalErrors: true);
On 2017/01/30 09:04:39, Johnni Winther wrote:
> Why suppress errors here?

Added:

      // We tokenize source files twice to keep memory usage low. This is the
      // second time, and the first time was in [buildOutline] above. So this
      // time we suppress lexical errors.

[ahe, 2017-02-01 12:43:45]

Patchset #4 (id:60001) has been deleted

[ahe, 2017-02-01 12:57:38]

Patchset #4 (id:80001) has been deleted

[ahe, 2017-02-01 12:59:17]

Description was changed from

==========
Port parser and scanner fixes from rasta branch.
==========

to

==========
Port parser and scanner fixes from rasta branch.

R=johnniwinther@google.com

Committed:
https://github.com/dart-lang/sdk/commit/ce742a1697661c55115bdb8009a3aee6febb16cb
==========

[ahe, 2017-02-01 12:59:18]

Committed patchset #4 (id:100001) manually as
ce742a1697661c55115bdb8009a3aee6febb16cb (presubmit successful).