Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(348)

Side by Side Diff: sdk/lib/_internal/compiler/implementation/scanner/string_scanner.dart

Issue 40583002: Incorporates feedback by Nicolas for UTF-8 bytes based scanner CL (Closed) Base URL: https://dart.googlecode.com/svn/branches/bleeding_edge/dart
Patch Set: Created 7 years, 1 month ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch | Annotate | Revision Log
OLDNEW
1 // Copyright (c) 2011, the Dart project authors. Please see the AUTHORS file 1 // Copyright (c) 2011, the Dart project authors. Please see the AUTHORS file
2 // for details. All rights reserved. Use of this source code is governed by a 2 // for details. All rights reserved. Use of this source code is governed by a
3 // BSD-style license that can be found in the LICENSE file. 3 // BSD-style license that can be found in the LICENSE file.
4 4
5 part of scanner; 5 part of scanner;
6 6
7 /** 7 /**
8 * Scanner that reads from a String and creates tokens that points to 8 * Scanner that reads from a String and creates tokens that points to
9 * substrings. 9 * substrings.
10 */ 10 */
(...skipping 24 matching lines...) Expand all
35 35
36 int advance() => string.codeUnitAt(++scanOffset); 36 int advance() => string.codeUnitAt(++scanOffset);
37 int peek() => string.codeUnitAt(scanOffset + 1); 37 int peek() => string.codeUnitAt(scanOffset + 1);
38 38
39 int get stringOffset => scanOffset; 39 int get stringOffset => scanOffset;
40 40
41 int currentAsUnicode(int next) => next; 41 int currentAsUnicode(int next) => next;
42 42
43 void handleUnicode(int startScanOffset) { } 43 void handleUnicode(int startScanOffset) { }
44 44
45
46 Token firstToken() => tokens.next; 45 Token firstToken() => tokens.next;
47 Token previousToken() => tail; 46 Token previousToken() => tail;
48 47
49 void appendSubstringToken(PrecedenceInfo info, int start, 48 void appendSubstringToken(PrecedenceInfo info, int start,
50 bool asciiOnly, [int extraOffset = 0]) { 49 bool asciiOnly, [int extraOffset = 0]) {
51 tail.next = new StringToken.fromSubstring(info, string, start, 50 tail.next = new StringToken.fromSubstring(info, string, start,
52 scanOffset + extraOffset, tokenStart, true); 51 scanOffset + extraOffset, tokenStart, canonicalize: true);
53 tail = tail.next; 52 tail = tail.next;
54 } 53 }
55 } 54 }
OLDNEW

Powered by Google App Engine
This is Rietveld 408576698