Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(751)

Unified Diff: src/lexer/lexer_py.re

Issue 80503002: Experimental lexer generator: Fix byte order mark detection. (Closed) Base URL: https://v8.googlecode.com/svn/branches/experimental/parser
Patch Set: Created 7 years, 1 month ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View side-by-side diff with in-line comments
Download patch
« no previous file with comments | « src/lexer/even-more-experimental-scanner.cc ('k') | tools/lexer_generator/code_generator.jinja » ('j') | no next file with comments »
Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
Index: src/lexer/lexer_py.re
diff --git a/src/lexer/lexer_py.re b/src/lexer/lexer_py.re
index 2ce0db54899469526637f52b81bc1ec5b4de2b82..b9c75d1caccb69fff8e13cde2886282be15dbb87 100644
--- a/src/lexer/lexer_py.re
+++ b/src/lexer/lexer_py.re
@@ -26,7 +26,8 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
whitespace_char = [:whitespace:];
-whitespace = whitespace_char+;
+byte_order_mark_char = [:byte_order_mark:];
+whitespace = whitespace_char+|byte_order_mark_char+;
identifier_start = [$_:letter:];
identifier_char = [:identifier_start::identifier_part_not_letter:];
line_terminator = [:line_terminator:];
« no previous file with comments | « src/lexer/even-more-experimental-scanner.cc ('k') | tools/lexer_generator/code_generator.jinja » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698