Implement ES6 unicode escapes.

Implement ES6 unicode escapes.

OBSOLETE; pls look at the split CLs instead:
part 1: https://codereview.chromium.org/716423002/

BUG=v8:3648
LOG=N

[marja, 2014-11-07 15:45:10]

marja@chromium.org changed reviewers:
+ rossberg@chromium.org

[marja, 2014-11-07 15:45:11]

rossberg, ptal

[arv (Not doing code reviews), 2014-11-07 16:23:19]

arv@chromium.org changed reviewers:
+ arv@chromium.org

[arv (Not doing code reviews), 2014-11-07 16:23:20]

Should we add a new flag for this?

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc
File src/scanner.cc (right):

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc#newcode901
src/scanner.cc:901: uc32 c = ScanHexNumber(4);
In case of u{ then there is no upper limit for the number of hex digits.

https://codereview.chromium.org/706263002/diff/20001/test/cctest/test-parsing.cc
File test/cctest/test-parsing.cc (right):

https://codereview.chromium.org/706263002/diff/20001/test/cctest/test-parsing...
test/cctest/test-parsing.cc:4316: "\"foob\\u{c4}r\"",
Can you add a test for 

http://people.mozilla.org/~jorendorff/es6-draft.html#sec-string-literals-stat...

UnicodeEscapeSequence :: u{ HexDigits }

  It is a Syntax Error if the MV of HexDigits > 1114111.

https://codereview.chromium.org/706263002/diff/20001/test/cctest/test-parsing...
test/cctest/test-parsing.cc:4344: "/[\\u{0062-\\u{0066 ]oo/",
Can you add a test with more than 4 hex numbers?

The following should be legal

\U{000000000000000000000000000000000000000000000000062}

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
File test/mjsunit/es6/unicode-escapes.js (right):

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
test/mjsunit/es6/unicode-escapes.js:9: function TestVariableNames1() {
Or

(function TestVariableNames1() {
   ...
})();

less likely to have typos in the name cause tests to not run.

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
test/mjsunit/es6/unicode-escapes.js:28: var s2 = "foob\u{0061}r";
var s2 = "foob\u{61}r";

[caitp (gmail), 2014-11-07 16:23:55]

caitpotter88@gmail.com changed reviewers:
+ caitpotter88@gmail.com

[caitp (gmail), 2014-11-07 16:23:55]

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc
File src/scanner.cc (right):

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc#newcode901
src/scanner.cc:901: uc32 c = ScanHexNumber(4);
If `has_brace` is true, we want the HexDigits non-terminal, which is not limited
to 4 digits (as I'm reading it) --- as opposed to `Hex4Digits` non-terminal
which should be exactly 4 digits.

I guess arv beat me to it :>

[mathias, 2014-11-07 16:37:57]

mathias@qiwi.be changed reviewers:
+ mathias@qiwi.be

[mathias, 2014-11-07 16:37:57]

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
File test/mjsunit/es6/unicode-escapes.js (right):

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
test/mjsunit/es6/unicode-escapes.js:34: // flags.
`u{…}` escapes are only allowed in regular expressions when the `u` flag is set
– they should throw otherwise.

[caitp (gmail), 2014-11-07 16:50:39]

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc
File src/scanner.cc (right):

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc#newcode907
src/scanner.cc:907: return 1;
shouldn't this return -1 so that the scanner knows it's a bad token? otherwise
it looks the same as \u{1}

[caitp (gmail), 2014-11-13 03:21:43]

https://codereview.chromium.org/706263002/diff/60001/src/scanner.cc
File src/scanner.cc (right):

https://codereview.chromium.org/706263002/diff/60001/src/scanner.cc#newcode908
src/scanner.cc:908: if (c0_ != 'u') return false;
Will cause `var \x` to look like `var \u{0}x`, if I'm not mistaken

[marja, 2014-11-13 10:06:23]

https://codereview.chromium.org/706263002/diff/60001/src/scanner.cc
File src/scanner.cc (right):

https://codereview.chromium.org/706263002/diff/60001/src/scanner.cc#newcode908
src/scanner.cc:908: if (c0_ != 'u') return false;
On 2014/11/13 03:21:42, caitp wrote:
> Will cause `var \x` to look like `var \u{0}x`, if I'm not mistaken

Oh, yes, this was a pure editing error.

[marja, 2014-11-13 10:07:57]

I'll split this CL up, since the regexp escape handling is more complicated
(requires checking that we have the 'u' flag). So in the first part I'll only
add the non-regexp \u{xxxx} cases behind a --harmony-unicode-escapes flag.

[rossberg, 2014-11-13 10:10:20]

On 2014/11/13 10:07:57, marja wrote:
> I'll split this CL up, since the regexp escape handling is more complicated
> (requires checking that we have the 'u' flag). So in the first part I'll only
> add the non-regexp \u{xxxx} cases behind a --harmony-unicode-escapes flag.

Sounds good. Nit: --harmony-unicode is good enough. :)

[marja, 2014-11-13 11:53:20]

I split the first part off of this CL, ptal here:
https://codereview.chromium.org/716423002/

(My intention was to do all the things you pointed out; except the ones related
to regexps.)

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc
File src/scanner.cc (right):

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc#newcode901
src/scanner.cc:901: uc32 c = ScanHexNumber(4);
On 2014/11/07 16:23:20, arv wrote:
> In case of u{ then there is no upper limit for the number of hex digits.

Done.

https://codereview.chromium.org/706263002/diff/20001/src/scanner.cc#newcode907
src/scanner.cc:907: return 1;
On 2014/11/07 16:50:39, caitp wrote:
> shouldn't this return -1 so that the scanner knows it's a bad token? otherwise
> it looks the same as \u{1}

Done.

https://codereview.chromium.org/706263002/diff/20001/test/cctest/test-parsing.cc
File test/cctest/test-parsing.cc (right):

https://codereview.chromium.org/706263002/diff/20001/test/cctest/test-parsing...
test/cctest/test-parsing.cc:4316: "\"foob\\u{c4}r\"",
On 2014/11/07 16:23:20, arv wrote:
> Can you add a test for 
> 
>
http://people.mozilla.org/~jorendorff/es6-draft.html#sec-string-literals-stat...
> 
> UnicodeEscapeSequence :: u{ HexDigits }
> 
>   It is a Syntax Error if the MV of HexDigits > 1114111.

Done.

https://codereview.chromium.org/706263002/diff/20001/test/cctest/test-parsing...
test/cctest/test-parsing.cc:4344: "/[\\u{0062-\\u{0066 ]oo/",
On 2014/11/07 16:23:20, arv wrote:
> Can you add a test with more than 4 hex numbers?
> 
> The following should be legal
> 
> \U{000000000000000000000000000000000000000000000000062}

Done.

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
File test/mjsunit/es6/unicode-escapes.js (right):

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
test/mjsunit/es6/unicode-escapes.js:9: function TestVariableNames1() {
On 2014/11/07 16:23:20, arv wrote:
> Or
> 
> (function TestVariableNames1() {
>    ...
> })();
> 
> less likely to have typos in the name cause tests to not run.

Done.

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
test/mjsunit/es6/unicode-escapes.js:28: var s2 = "foob\u{0061}r";
On 2014/11/07 16:23:20, arv wrote:
> var s2 = "foob\u{61}r";

Done.

https://codereview.chromium.org/706263002/diff/20001/test/mjsunit/es6/unicode...
test/mjsunit/es6/unicode-escapes.js:34: // flags.
On 2014/11/07 16:37:57, mathias wrote:
> `u{…}` escapes are only allowed in regular expressions when the `u` flag is
set
> – they should throw otherwise.

Regexp escapes will be added in a separate CL.