Update unicode to 7.0.0.

Update unicode to 7.0.0.

And do not use code points with PATTERN_* property for identifier start.
Maintain that \u180E is a white space character.

BUG=v8:2892
LOG=Y
R=dpino@igalia.com, mathias@qiwi.be

Committed: https://code.google.com/p/v8/source/detail?r=24473

[Yang, 2014-10-07 15:21:25]

yangguo@chromium.org changed reviewers:
+ dpino@igalia.com, mathias@qiwi.be, rossberg@chromium.org

[Yang, 2014-10-07 15:21:25]

Please take a look. The tables have been generated using a script in V8's
internal repository.

[mathias, 2014-10-07 15:38:11]

On 2014/10/07 15:21:25, Yang wrote:
> Please take a look. The tables have been generated using a script in V8's
> internal repository.

You may want to add tests to make sure `ID_Start` matches these:

* U+2118
* U+212E
* U+309B
* U+309C

…and `ID_Continue` matches these:

* U+00B7
* U+0387
* U+1369
* U+1370
* U+1371
* U+19DA

Those are the code points that match the respective property according to
`DerivedCoreProperties.txt`, yet don’t match these properties if you’re
adding/removing the categories manually based on the property definition in
TR31.

[Diego Pino, 2014-10-07 15:57:38]

On 2014/10/07 15:21:25, Yang wrote:
> Please take a look. The tables have been generated using a script in V8's
> internal repository.

I'm not a reviewer but the patch LGTM. There's one thing I don't understand
though and it's why renaming IdentifierPart as IdentifierContinue. ES6 describes
an IdentifierName as:
(http://people.mozilla.org/~jorendorff/es6-draft.html#sec-identifier-names)

IdentifierName ::
   IdentifierStart
   IdentifierName IdentifierPart

IdentifierPart is referenced in other sections of the spec. For clarity I would
keep the same nomenclature.

[mathias, 2014-10-07 16:26:59]

I’m slightly confused by this patch. Its description is “update to Unicode
v7.0.0” but the intent also seems to be to update to the ES6 definition of
`Identifier`. However, the current patch only does that partially, as it doesn’t
support supplementary Unicode symbols. E.g. `eval('var 𐐼')` shouldn’t throw an
error (`𐐼` is U+1043C DESERET SMALL LETTER DEE).

[Yang, 2014-10-07 17:50:42]

On 2014/10/07 16:26:59, mathias wrote:
> I’m slightly confused by this patch. Its description is “update to Unicode
> v7.0.0” but the intent also seems to be to update to the ES6 definition of
> `Identifier`. However, the current patch only does that partially, as it
doesn’t
> support supplementary Unicode symbols. E.g. `eval('var 𐐼')` shouldn’t throw an
> error (`𐐼` is U+1043C DESERET SMALL LETTER DEE).

I'm not particular familiar witb the spec changes in this area. I merely reran
the script to generate V8's Unicode tables with the data files for Unicode
7.0.0. Thereby I used the same rules to generate the predicates, except that I
added a predicate Pattern for the properties PATTERN_* to correctly implement
ID_Start and ID_Continue. I'm not sure how your example is related and where it
is described in either ES6 or Unicode. Can you give me a tip?

[mathias, 2014-10-07 20:24:10]

On 2014/10/07 17:50:42, Yang wrote:
> On 2014/10/07 16:26:59, mathias wrote:
> > I’m slightly confused by this patch. Its description is “update to Unicode
> > v7.0.0” but the intent also seems to be to update to the ES6 definition of
> > `Identifier`. However, the current patch only does that partially, as it
> doesn’t
> > support supplementary Unicode symbols. E.g. `eval('var 𐐼')` shouldn’t throw
an
> > error (`𐐼` is U+1043C DESERET SMALL LETTER DEE).
> 
> I'm not particular familiar witb the spec changes in this area. I merely reran
> the script to generate V8's Unicode tables with the data files for Unicode
> 7.0.0. Thereby I used the same rules to generate the predicates, except that I
> added a predicate Pattern for the properties PATTERN_* to correctly implement
> ID_Start and ID_Continue. I'm not sure how your example is related and where
it
> is described in either ES6 or Unicode. Can you give me a tip?

ES6 allows supplementary characters (i.e. characters with code point > 0xFFFF)
in identifiers, while ES5 doesn’t. See
https://people.mozilla.org/~jorendorff/es6-draft.html#sec-names-and-keywords,
and note how `IdentifierStart` refers to `UnicodeIDStart` which is defined as
‘any Unicode code point with the Unicode property “ID_Start” or
“Other_ID_Start”’ — this doesn’t exclude supplementary code points. But ES5 did:
https://bugs.ecmascript.org/show_bug.cgi?id=469#c0 The same thing goes for
`IdentifierPart` and `UnicodeIDContinue`.

To illustrate the difference in code: compare
https://gist.github.com/mathiasbynens/6334847#file-javascript-identifier-rege...
(ES5) with
https://gist.github.com/mathiasbynens/6334847#file-javascript-identifier-rege...
(ES6).

[Yang, 2014-10-08 06:32:16]

On 2014/10/07 20:24:10, mathias wrote:
> On 2014/10/07 17:50:42, Yang wrote:
> > On 2014/10/07 16:26:59, mathias wrote:
> > > I’m slightly confused by this patch. Its description is “update to Unicode
> > > v7.0.0” but the intent also seems to be to update to the ES6 definition of
> > > `Identifier`. However, the current patch only does that partially, as it
> > doesn’t
> > > support supplementary Unicode symbols. E.g. `eval('var 𐐼')` shouldn’t
throw
> an
> > > error (`𐐼` is U+1043C DESERET SMALL LETTER DEE).
> > 
> > I'm not particular familiar witb the spec changes in this area. I merely
reran
> > the script to generate V8's Unicode tables with the data files for Unicode
> > 7.0.0. Thereby I used the same rules to generate the predicates, except that
I
> > added a predicate Pattern for the properties PATTERN_* to correctly
implement
> > ID_Start and ID_Continue. I'm not sure how your example is related and where
> it
> > is described in either ES6 or Unicode. Can you give me a tip?
> 
> ES6 allows supplementary characters (i.e. characters with code point > 0xFFFF)
> in identifiers, while ES5 doesn’t. See
> https://people.mozilla.org/~jorendorff/es6-draft.html#sec-names-and-keywords,
> and note how `IdentifierStart` refers to `UnicodeIDStart` which is defined as
> ‘any Unicode code point with the Unicode property “ID_Start” or
> “Other_ID_Start”’ — this doesn’t exclude supplementary code points. But ES5
did:
> https://bugs.ecmascript.org/show_bug.cgi?id=469#c0 The same thing goes for
> `IdentifierPart` and `UnicodeIDContinue`.
> 
> To illustrate the difference in code: compare
>
https://gist.github.com/mathiasbynens/6334847#file-javascript-identifier-rege...
> (ES5) with
>
https://gist.github.com/mathiasbynens/6334847#file-javascript-identifier-rege...
> (ES6).

I see. This seems like a somewhat separate issue than the one here. In this CL I
set out to solve issue 2892, by creating a new predicate generated from Unicode
data, and while I was at it, I also updated that data to 7.0.0.

I'll file a bug for the supplementary characters issue and leave that to later.

I'll update this CL later according to the comments above.

[Yang, 2014-10-08 13:20:57]

On 2014/10/08 06:32:16, Yang wrote:
> On 2014/10/07 20:24:10, mathias wrote:
> > On 2014/10/07 17:50:42, Yang wrote:
> > > On 2014/10/07 16:26:59, mathias wrote:
> > > > I’m slightly confused by this patch. Its description is “update to
Unicode
> > > > v7.0.0” but the intent also seems to be to update to the ES6 definition
of
> > > > `Identifier`. However, the current patch only does that partially, as it
> > > doesn’t
> > > > support supplementary Unicode symbols. E.g. `eval('var 𐐼')` shouldn’t
> throw
> > an
> > > > error (`𐐼` is U+1043C DESERET SMALL LETTER DEE).
> > > 
> > > I'm not particular familiar witb the spec changes in this area. I merely
> reran
> > > the script to generate V8's Unicode tables with the data files for Unicode
> > > 7.0.0. Thereby I used the same rules to generate the predicates, except
that
> I
> > > added a predicate Pattern for the properties PATTERN_* to correctly
> implement
> > > ID_Start and ID_Continue. I'm not sure how your example is related and
where
> > it
> > > is described in either ES6 or Unicode. Can you give me a tip?
> > 
> > ES6 allows supplementary characters (i.e. characters with code point >
0xFFFF)
> > in identifiers, while ES5 doesn’t. See
> >
https://people.mozilla.org/~jorendorff/es6-draft.html#sec-names-and-keywords,
> > and note how `IdentifierStart` refers to `UnicodeIDStart` which is defined
as
> > ‘any Unicode code point with the Unicode property “ID_Start” or
> > “Other_ID_Start”’ — this doesn’t exclude supplementary code points. But ES5
> did:
> > https://bugs.ecmascript.org/show_bug.cgi?id=469#c0 The same thing goes for
> > `IdentifierPart` and `UnicodeIDContinue`.
> > 
> > To illustrate the difference in code: compare
> >
>
https://gist.github.com/mathiasbynens/6334847#file-javascript-identifier-rege...
> > (ES5) with
> >
>
https://gist.github.com/mathiasbynens/6334847#file-javascript-identifier-rege...
> > (ES6).
> 
> I see. This seems like a somewhat separate issue than the one here. In this CL
I
> set out to solve issue 2892, by creating a new predicate generated from
Unicode
> data, and while I was at it, I also updated that data to 7.0.0.
> 
> I'll file a bug for the supplementary characters issue and leave that to
later.
> 
> I'll update this CL later according to the comments above.

- Updated to handle Other_ID_Start and Other_ID_Continue correctly.
- Added unittests.
- Refactored parts of src/unicode.* to src/unicode-decoder.h

[mathias, 2014-10-08 13:55:47]

LGTM.

Nit: inconsistent use of uppercase/lowercase for hex digits (e.g. `\u180e` vs.
`0x180E` in the same file).

https://codereview.chromium.org/638643002/diff/40001/test/mjsunit/var.js
File test/mjsunit/var.js (right):

https://codereview.chromium.org/638643002/diff/40001/test/mjsunit/var.js#newc...
test/mjsunit/var.js:40: assertThrows("var \\u2E2F;", SyntaxError);
Maybe add the tests from https://codereview.chromium.org/638643002/#msg3 here as
well, just in case? Or do you think it’s enough to have them in
`unicode-predicates-unittest.cc` only?

[mathias, 2014-10-08 14:01:12]

Re: U+180E, it seems a bit strange to use Unicode v7.0.0 for one part but
Unicode v5.1.0 for other parts of the engine. Just to be clear, the ES spec
allows using a more recent version than v5.1.0:
https://people.mozilla.org/~jorendorff/es6-draft.html#sec-conformance I wonder
if it’d be less confusing to pick one or the other rather than kind-of going
with both at the same time.

[Yang, 2014-10-08 14:06:45]

On 2014/10/08 14:01:12, mathias wrote:
> Re: U+180E, it seems a bit strange to use Unicode v7.0.0 for one part but
> Unicode v5.1.0 for other parts of the engine. Just to be clear, the ES spec
> allows using a more recent version than v5.1.0:
> https://people.mozilla.org/~jorendorff/es6-draft.html#sec-conformance I wonder
> if it’d be less confusing to pick one or the other rather than kind-of going
> with both at the same time.

Addressed comments.

Regarding U+180E, I'm relying on ES6 draft section 11.2
"ECMAScript implementations must recognize as Whitespace code points listed in
the “Separator, space” (Zs) category by Unicode 5.1."

There seems not much wriggle room here. It does not state "Unicode 5.1 or later"
here.

[mathias, 2014-10-08 14:15:20]

On 2014/10/08 14:06:45, Yang wrote:
> Regarding U+180E, I'm relying on ES6 draft section 11.2
> "ECMAScript implementations must recognize as Whitespace code points listed in
> the “Separator, space” (Zs) category by Unicode 5.1."
> 
> There seems not much wriggle room here. It does not state "Unicode 5.1 or
later"
> here.

Ah, right: it says “ECMAScript implementations may also recognize as Whitespace
*additional* category Zs code points from subsequent editions of the Unicode
Standard” (emphasis mine) but it doesn’t say anything about removing code points
from the set. Thanks for pointing this out!

[Yang, 2014-10-08 14:55:23]

Committed patchset #4 (id:60001) manually as 24473 (presubmit successful).