Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(3)

Issues Search
    My Issues | Starred     Open | Closed | All

Side by Side Diff: src/lexer/lexer_py.re

Issue 62103017: Experimental parser: rule grammar refactor (Closed) Base URL: https://v8.googlecode.com/svn/branches/experimental/parser
Patch Set: Created 7 years, 1 month ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch | Annotate | Revision Log
« no previous file with comments | « no previous file | tools/lexer_generator/action_test.py » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Copyright 2013 the V8 project authors. All rights reserved. 1 # Copyright 2013 the V8 project authors. All rights reserved.
2 # Redistribution and use in source and binary forms, with or without 2 # Redistribution and use in source and binary forms, with or without
3 # modification, are permitted provided that the following conditions are 3 # modification, are permitted provided that the following conditions are
4 # met: 4 # met:
5 # 5 #
6 # * Redistributions of source code must retain the above copyright 6 # * Redistributions of source code must retain the above copyright
7 # notice, this list of conditions and the following disclaimer. 7 # notice, this list of conditions and the following disclaimer.
8 # * Redistributions in binary form must reproduce the above 8 # * Redistributions in binary form must reproduce the above
9 # copyright notice, this list of conditions and the following 9 # copyright notice, this list of conditions and the following
10 # disclaimer in the documentation and/or other materials provided 10 # disclaimer in the documentation and/or other materials provided
11 # with the distribution. 11 # with the distribution.
12 # * Neither the name of Google Inc. nor the names of its 12 # * Neither the name of Google Inc. nor the names of its
13 # contributors may be used to endorse or promote products derived 13 # contributors may be used to endorse or promote products derived
14 # from this software without specific prior written permission. 14 # from this software without specific prior written permission.
15 # 15 #
16 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 16 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
17 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 17 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
18 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 18 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
19 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 19 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
20 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 20 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
21 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 21 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
22 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 22 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 23 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 27
28 whitespace_char = [ \t\v\f\r:ws:\240]; 28 whitespace_char = [ \t\v\f\r:ws:\240];
29 whitespace = whitespace_char+; 29 whitespace = whitespace_char+;
30 identifier_start = [$_a-zA-Z:lit:]; # TODO add relevant latin1 char codes 30 identifier_start = [$_a-zA-Z:lit:];
31 identifier_char = [0-9:identifier_start:]; 31 identifier_char = [0-9:identifier_start:];
32 line_terminator = [\n\r]; 32 line_terminator = [\n\r];
33 digit = [0-9]; 33 digit = [0-9];
34 hex_digit = [0-9a-fA-F]; 34 hex_digit = [0-9a-fA-F];
35 maybe_exponent = ([eE] [\-+]? digit+)?; 35 maybe_exponent = /([eE][\-+]?[:digit:]+)?/;
36 number = ("0x" hex_digit+) | (("." digit+ maybe_exponent) | (digit+ ("." digit*) ? maybe_exponent)); 36 number =
37 /0x[:hex_digit:]+/ | (
38 /\.[:digit:]+/ maybe_exponent |
39 /[:digit:]+(\.[:digit:]*)?/ maybe_exponent );
37 40
38 <default> 41 # grammar is
39 "|=" push_token(ASSIGN_BIT_OR) 42 # regex <action_on_state_entry|action_on_match|transition>
40 "^=" push_token(ASSIGN_BIT_XOR) 43 #
41 "&=" push_token(ASSIGN_BIT_AND) 44 # actions can be c code enclosed in {} or identifiers to be passed to codegen
42 "+=" push_token(ASSIGN_ADD) 45 # transition must be in continue or the name of a subgraph
43 "-=" push_token(ASSIGN_SUB)
44 "*=" push_token(ASSIGN_MUL)
45 "/=" push_token(ASSIGN_DIV)
46 "%=" push_token(ASSIGN_MOD)
47 46
48 "===" push_token(EQ_STRICT) 47 <<default>>
49 "==" push_token(EQ) 48 "|=" <|push_token(ASSIGN_BIT_OR)|>
50 "=" push_token(ASSIGN) 49 "^=" <|push_token(ASSIGN_BIT_XOR)|>
51 "!==" push_token(NE_STRICT) 50 "&=" <|push_token(ASSIGN_BIT_AND)|>
52 "!=" push_token(NE) 51 "+=" <|push_token(ASSIGN_ADD)|>
53 "!" push_token(NOT) 52 "-=" <|push_token(ASSIGN_SUB)|>
53 "*=" <|push_token(ASSIGN_MUL)|>
54 "/=" <|push_token(ASSIGN_DIV)|>
55 "%=" <|push_token(ASSIGN_MOD)|>
54 56
55 "//" <<SingleLineComment>> 57 "===" <|push_token(EQ_STRICT)|>
56 "/*" <<MultiLineComment>> 58 "==" <|push_token(EQ)|>
57 "<!--" <<HtmlComment>> 59 "=" <|push_token(ASSIGN)|>
60 "!==" <|push_token(NE_STRICT)|>
61 "!=" <|push_token(NE)|>
62 "!" <|push_token(NOT)|>
63
64 "//" <||SingleLineComment>
65 "/*" <||MultiLineComment>
66 "<!--" <||HtmlComment>
58 67
59 #whitespace* "-->" { if (just_seen_line_terminator_) { YYSETCONDITION(kCondition SingleLineComment); goto yyc_SingleLineComment; } else { --cursor_; send(Token:: DEC); start_ = cursor_; goto yyc_Normal; } } 68 #whitespace* "-->" { if (just_seen_line_terminator_) { YYSETCONDITION(kCondition SingleLineComment); goto yyc_SingleLineComment; } else { --cursor_; send(Token:: DEC); start_ = cursor_; goto yyc_Normal; } }
60 69
61 ">>>=" push_token(ASSIGN_SHR) 70 ">>>=" <|push_token(ASSIGN_SHR)|>
62 ">>>" push_token(SHR) 71 ">>>" <|push_token(SHR)|>
63 "<<=" push_token(ASSIGN_SHL) 72 "<<=" <|push_token(ASSIGN_SHL)|>
64 ">>=" push_token(ASSIGN_SAR) 73 ">>=" <|push_token(ASSIGN_SAR)|>
65 "<=" push_token(LTE) 74 "<=" <|push_token(LTE)|>
66 ">=" push_token(GTE) 75 ">=" <|push_token(GTE)|>
67 "<<" push_token(SHL) 76 "<<" <|push_token(SHL)|>
68 ">>" push_token(SAR) 77 ">>" <|push_token(SAR)|>
69 "<" push_token(LT) 78 "<" <|push_token(LT)|>
70 ">" push_token(GT) 79 ">" <|push_token(GT)|>
71 80
72 number push_token(NUMBER) 81 number <|push_token(NUMBER)|>
73 # number identifier_char push_token(ILLEGAL) 82 # is this necessary?
83 number identifier_char <|push_token(ILLEGAL)|>
74 84
75 "(" push_token(LPAREN) 85 "(" <|push_token(LPAREN)|>
76 ")" push_token(RPAREN) 86 ")" <|push_token(RPAREN)|>
77 "[" push_token(LBRACK) 87 "[" <|push_token(LBRACK)|>
78 "]" push_token(RBRACK) 88 "]" <|push_token(RBRACK)|>
79 "{" push_token(LBRACE) 89 "{" <|push_token(LBRACE)|>
80 "}" push_token(RBRACE) 90 "}" <|push_token(RBRACE)|>
81 ":" push_token(COLON) 91 ":" <|push_token(COLON)|>
82 ";" push_token(SEMICOLON) 92 ";" <|push_token(SEMICOLON)|>
83 "." push_token(PERIOD) 93 "." <|push_token(PERIOD)|>
84 "?" push_token(CONDITIONAL) 94 "?" <|push_token(CONDITIONAL)|>
85 "++" push_token(INC) 95 "++" <|push_token(INC)|>
86 "--" push_token(DEC) 96 "--" <|push_token(DEC)|>
87 97
88 "||" push_token(OR) 98 "||" <|push_token(OR)|>
89 "&&" push_token(AND) 99 "&&" <|push_token(AND)|>
90 100
91 "|" push_token(BIT_OR) 101 "|" <|push_token(BIT_OR)|>
92 "^" push_token(BIT_XOR) 102 "^" <|push_token(BIT_XOR)|>
93 "&" push_token(BIT_AND) 103 "&" <|push_token(BIT_AND)|>
94 "+" push_token(ADD) 104 "+" <|push_token(ADD)|>
95 "-" push_token(SUB) 105 "-" <|push_token(SUB)|>
96 "*" push_token(MUL) 106 "*" <|push_token(MUL)|>
97 "/" push_token(DIV) 107 "/" <|push_token(DIV)|>
98 "%" push_token(MOD) 108 "%" <|push_token(MOD)|>
99 "~" push_token(BIT_NOT) 109 "~" <|push_token(BIT_NOT)|>
100 "," push_token(COMMA) 110 "," <|push_token(COMMA)|>
101 111
102 line_terminator+ { PUSH_LINE_TERMINATOR(); } 112 line_terminator+ <|push_line_terminator|>
103 whitespace <<skip>> 113 whitespace <|skip|>
104 114
105 "\"" <<DoubleQuoteString>> 115 "\"" <||DoubleQuoteString>
106 "'" <<SingleQuoteString>> 116 "'" <||SingleQuoteString>
107 117
108 # all keywords 118 # all keywords
109 "break" push_token(BREAK) 119 "break" <|push_token(BREAK)|>
110 "case" push_token(CASE) 120 "case" <|push_token(CASE)|>
111 "catch" push_token(CATCH) 121 "catch" <|push_token(CATCH)|>
112 "class" push_token(FUTURE_RESERVED_WORD) 122 "class" <|push_token(FUTURE_RESERVED_WORD)|>
113 "const" push_token(CONST) 123 "const" <|push_token(CONST)|>
114 "continue" push_token(CONTINUE) 124 "continue" <|push_token(CONTINUE)|>
115 "debugger" push_token(DEBUGGER) 125 "debugger" <|push_token(DEBUGGER)|>
116 "default" push_token(DEFAULT) 126 "default" <|push_token(DEFAULT)|>
117 "delete" push_token(DELETE) 127 "delete" <|push_token(DELETE)|>
118 "do" push_token(DO) 128 "do" <|push_token(DO)|>
119 "else" push_token(ELSE) 129 "else" <|push_token(ELSE)|>
120 "enum" push_token(FUTURE_RESERVED_WORD) 130 "enum" <|push_token(FUTURE_RESERVED_WORD)|>
121 "export" push_token(FUTURE_RESERVED_WORD) 131 "export" <|push_token(FUTURE_RESERVED_WORD)|>
122 "extends" push_token(FUTURE_RESERVED_WORD) 132 "extends" <|push_token(FUTURE_RESERVED_WORD)|>
123 "false" push_token(FALSE_LITERAL) 133 "false" <|push_token(FALSE_LITERAL)|>
124 "finally" push_token(FINALLY) 134 "finally" <|push_token(FINALLY)|>
125 "for" push_token(FOR) 135 "for" <|push_token(FOR)|>
126 "function" push_token(FUNCTION) 136 "function" <|push_token(FUNCTION)|>
127 "if" push_token(IF) 137 "if" <|push_token(IF)|>
128 "implements" push_token(FUTURE_STRICT_RESERVED_WORD) 138 "implements" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
129 "import" push_token(FUTURE_RESERVED_WORD) 139 "import" <|push_token(FUTURE_RESERVED_WORD)|>
130 "in" push_token(IN) 140 "in" <|push_token(IN)|>
131 "instanceof" push_token(INSTANCEOF) 141 "instanceof" <|push_token(INSTANCEOF)|>
132 "interface" push_token(FUTURE_STRICT_RESERVED_WORD) 142 "interface" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
133 "let" push_token(FUTURE_STRICT_RESERVED_WORD) 143 "let" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
134 "new" push_token(NEW) 144 "new" <|push_token(NEW)|>
135 "null" push_token(NULL_LITERAL) 145 "null" <|push_token(NULL_LITERAL)|>
136 "package" push_token(FUTURE_STRICT_RESERVED_WORD) 146 "package" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
137 "private" push_token(FUTURE_STRICT_RESERVED_WORD) 147 "private" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
138 "protected" push_token(FUTURE_STRICT_RESERVED_WORD) 148 "protected" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
139 "public" push_token(FUTURE_STRICT_RESERVED_WORD) 149 "public" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
140 "return" push_token(RETURN) 150 "return" <|push_token(RETURN)|>
141 "static" push_token(FUTURE_STRICT_RESERVED_WORD) 151 "static" <|push_token(FUTURE_STRICT_RESERVED_WORD)|>
142 "super" push_token(FUTURE_RESERVED_WORD) 152 "super" <|push_token(FUTURE_RESERVED_WORD)|>
143 "switch" push_token(SWITCH) 153 "switch" <|push_token(SWITCH)|>
144 "this" push_token(THIS) 154 "this" <|push_token(THIS)|>
145 "throw" push_token(THROW) 155 "throw" <|push_token(THROW)|>
146 "true" push_token(TRUE_LITERAL) 156 "true" <|push_token(TRUE_LITERAL)|>
147 "try" push_token(TRY) 157 "try" <|push_token(TRY)|>
148 "typeof" push_token(TYPEOF) 158 "typeof" <|push_token(TYPEOF)|>
149 "var" push_token(VAR) 159 "var" <|push_token(VAR)|>
150 "void" push_token(VOID) 160 "void" <|push_token(VOID)|>
151 "while" push_token(WHILE) 161 "while" <|push_token(WHILE)|>
152 "with" push_token(WITH) 162 "with" <|push_token(WITH)|>
153 "yield" push_token(YIELD) 163 "yield" <|push_token(YIELD)|>
154 164
155 identifier_start push_token(IDENTIFIER) <<Identifier>> 165 identifier_start <|push_token(IDENTIFIER)|Identifier>
156 /\\u[0-9a-fA-F]{4}/ { 166 /\\u[0-9a-fA-F]{4}/ <{
157 if (V8_UNLIKELY(!ValidIdentifierStart())) { 167 if (V8_UNLIKELY(!ValidIdentifierStart())) {
158 PUSH_TOKEN(Token::ILLEGAL); 168 PUSH_TOKEN(Token::ILLEGAL);
169 // need to goto something here
159 } 170 }
160 } <<Identifier>> 171 }|push_token(IDENTIFIER)|Identifier>
161 172
162 eof <<terminate>> 173 eof <|terminate|>
163 default_action push_token(ILLEGAL) 174 default_action <push_token(ILLEGAL)>
164 175
165 <DoubleQuoteString> 176 <<DoubleQuoteString>>
166 /\\\n\r?/ <<continue>> 177 /\\\n\r?/ <||continue>
167 /\\\r\n?/ <<continue>> 178 /\\\r\n?/ <||continue>
168 /\\./ <<continue>> 179 /\\./ <||continue>
169 /\n|\r/ push_token(ILLEGAL) 180 /\n|\r/ <|push_token(ILLEGAL)|>
170 "\"" push_token(STRING) 181 "\"" <|push_token(STRING)|>
171 eof <<terminate_illegal>> 182 eof <|terminate_illegal|>
172 catch_all <<continue>> 183 catch_all <||continue>
173 184
174 <SingleQuoteString> 185 <<SingleQuoteString>>
175 /\\\n\r?/ <<continue>> 186 /\\\n\r?/ <||continue>
176 /\\\r\n?/ <<continue>> 187 /\\\r\n?/ <||continue>
177 /\\./ <<continue>> 188 /\\./ <||continue>
178 /\n|\r/ push_token(ILLEGAL) 189 /\n|\r/ <|push_token(ILLEGAL)|>
179 "'" push_token(STRING) 190 "'" <|push_token(STRING)|>
180 eof <<terminate_illegal>> 191 eof <|terminate_illegal|>
181 catch_all <<continue>> 192 catch_all <||continue>
182 193
183 <Identifier> 194 <<Identifier>>
184 identifier_char push_token(IDENTIFIER) <<continue>> 195 identifier_char <|push_token(IDENTIFIER)|continue>
185 /\\u[0-9a-fA-F]{4}/ { 196 /\\u[0-9a-fA-F]{4}/ <{
186 if (V8_UNLIKELY(!ValidIdentifierStart())) { 197 if (V8_UNLIKELY(!ValidIdentifierStart())) {
187 PUSH_TOKEN(Token::ILLEGAL); 198 PUSH_TOKEN(Token::ILLEGAL);
199 // need to goto something here
188 } 200 }
189 } <<continue>> 201 }|push_token(IDENTIFIER)|continue>
190 202
191 <SingleLineComment> 203 <<SingleLineComment>>
192 line_terminator { PUSH_LINE_TERMINATOR(); } 204 line_terminator <|push_line_terminator|>
193 catch_all <<continue>> 205 catch_all <||continue>
194 206
195 <MultiLineComment> 207 <<MultiLineComment>>
196 "*/" <<skip>> 208 "*/" <|skip|>
197 /\*[^\/]/ <<continue>> 209 # TODO find a way to generate the below rule
198 line_terminator { PUSH_LINE_TERMINATOR(); } <<continue>> 210 /\*[^\/]/ <||continue>
199 catch_all <<continue>> 211 line_terminator <|push_line_terminator|continue>
212 catch_all <||continue>
200 213
201 <HtmlComment> 214 <<HtmlComment>>
202 "-->" <<skip>> 215 "-->" <|skip|>
203 /--./ <<continue>> 216 # TODO find a way to generate the below rules
204 /-./ <<continue>> 217 /--./ <||continue>
205 line_terminator { PUSH_LINE_TERMINATOR(); } <<continue>> 218 /-./ <||continue>
206 catch_all <<continue>> 219 line_terminator <|push_line_terminator|continue>
220 catch_all <||continue>
OLDNEW
« no previous file with comments | « no previous file | tools/lexer_generator/action_test.py » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698