Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(103)

Side by Side Diff: runtime/vm/scanner.h

Issue 23452043: Introduce newline tokens into the token stream. (Closed) Base URL: https://dart.googlecode.com/svn/branches/bleeding_edge/dart
Patch Set: rebase (fingerprints) Created 7 years, 3 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch | Annotate | Revision Log
« no previous file with comments | « runtime/vm/parser.cc ('k') | runtime/vm/scanner.cc » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 // Copyright (c) 2011, the Dart project authors. Please see the AUTHORS file 1 // Copyright (c) 2011, the Dart project authors. Please see the AUTHORS file
2 // for details. All rights reserved. Use of this source code is governed by a 2 // for details. All rights reserved. Use of this source code is governed by a
3 // BSD-style license that can be found in the LICENSE file. 3 // BSD-style license that can be found in the LICENSE file.
4 // 4 //
5 // Scanner class for the Dart language. The scanner reads source text 5 // Scanner class for the Dart language. The scanner reads source text
6 // and produces a stream of tokens which is used by the parser. 6 // and produces a stream of tokens which is used by the parser.
7 // 7 //
8 8
9 #ifndef VM_SCANNER_H_ 9 #ifndef VM_SCANNER_H_
10 #define VM_SCANNER_H_ 10 #define VM_SCANNER_H_
(...skipping 185 matching lines...) Expand 10 before | Expand all | Expand 10 after
196 } 196 }
197 197
198 // Reads a number literal. 198 // Reads a number literal.
199 void ScanNumber(bool dec_point_seen); 199 void ScanNumber(bool dec_point_seen);
200 200
201 void ScanScriptTag(); 201 void ScanScriptTag();
202 202
203 static void PrintTokens(const GrowableTokenStream& ts); 203 static void PrintTokens(const GrowableTokenStream& ts);
204 204
205 TokenDescriptor current_token_; // Current token. 205 TokenDescriptor current_token_; // Current token.
206 TokenDescriptor newline_token_; // Newline token.
206 const String& source_; // The source text being tokenized. 207 const String& source_; // The source text being tokenized.
207 intptr_t source_length_; // The length of the source text. 208 intptr_t source_length_; // The length of the source text.
208 intptr_t lookahead_pos_; // Position of lookahead character 209 intptr_t lookahead_pos_; // Position of lookahead character
209 // within source_. 210 // within source_.
210 intptr_t token_start_; // Begin of current token in src_. 211 intptr_t token_start_; // Begin of current token in src_.
211 int32_t c0_; // Lookahead character. 212 int32_t c0_; // Lookahead character.
212 bool newline_seen_; // Newline before current token. 213 bool newline_seen_; // Newline before current token.
214 intptr_t prev_token_line_; // Line number of the previous token.
213 215
214 // The following fields keep track whether we are scanning a string literal 216 // The following fields keep track whether we are scanning a string literal
215 // and its interpolated expressions. 217 // and its interpolated expressions.
216 ScanContext* saved_context_; 218 ScanContext* saved_context_;
217 int32_t string_delimiter_; 219 int32_t string_delimiter_;
218 bool string_is_multiline_; 220 bool string_is_multiline_;
219 int brace_level_; 221 int brace_level_;
220 222
221 const String& private_key_; 223 const String& private_key_;
222 224
223 SourcePosition c0_pos_; // Source position of lookahead character c0_. 225 SourcePosition c0_pos_; // Source position of lookahead character c0_.
224 KeywordTable keywords_[Token::numKeywords]; 226 KeywordTable keywords_[Token::numKeywords];
225 Array& keyword_symbol_table_; // Access to keyword symbols in object store. 227 Array& keyword_symbol_table_; // Access to keyword symbols in object store.
226 }; 228 };
227 229
228 230
229 } // namespace dart 231 } // namespace dart
230 232
231 #endif // VM_SCANNER_H_ 233 #endif // VM_SCANNER_H_
OLDNEW
« no previous file with comments | « runtime/vm/parser.cc ('k') | runtime/vm/scanner.cc » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698