Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(984)

Unified Diff: src/lexer/lexer_py.re

Issue 80263003: Experimental parser: split classes into latin1 and non latin1 (Closed) Base URL: https://v8.googlecode.com/svn/branches/experimental/parser
Patch Set: Created 7 years, 1 month ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View side-by-side diff with in-line comments
Download patch
« no previous file with comments | « no previous file | tools/lexer_generator/code_generator.jinja » ('j') | no next file with comments »
Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
Index: src/lexer/lexer_py.re
diff --git a/src/lexer/lexer_py.re b/src/lexer/lexer_py.re
index 9644b4443a4f590a9684527b5f5fabf0d67fa364..2ce0db54899469526637f52b81bc1ec5b4de2b82 100644
--- a/src/lexer/lexer_py.re
+++ b/src/lexer/lexer_py.re
@@ -25,11 +25,11 @@
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-whitespace_char = [ \t\v\f\r:whitespace:\240];
+whitespace_char = [:whitespace:];
whitespace = whitespace_char+;
-identifier_start = [$_a-zA-Z:letter:];
-identifier_char = [0-9:identifier_part_not_letter::identifier_start:];
-line_terminator = [\n\r];
+identifier_start = [$_:letter:];
+identifier_char = [:identifier_start::identifier_part_not_letter:];
+line_terminator = [:line_terminator:];
digit = [0-9];
hex_digit = [0-9a-fA-F];
single_escape_char = ['"\\bfnrtv];
@@ -38,8 +38,7 @@ number =
/0[xX][:hex_digit:]+/ | (
/\.[:digit:]+/ maybe_exponent |
/[:digit:]+(\.[:digit:]*)?/ maybe_exponent );
-# TODO this is incomplete/incorrect
-line_terminator_sequence = (/\n\r?/)|(/\r\n?/);
+line_terminator_sequence = /[:line_terminator:]|\r\n/;
eos = [:eos:];
# grammar is
« no previous file with comments | « no previous file | tools/lexer_generator/code_generator.jinja » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698