Case folding mappings

- Added case folding mappings to unibrow.  They are currently affected
  by bug 149, as are our other mappings, but should be correct for all
  letters that don't case-fold more than 2^11 code points away.
- Also, took the opportunity to remove some unused predicates.

[Lasse Reichstein, 2008-11-17 07:35:33]

LGTM.
I haven't tried to understand the automatically generated code.

http://codereview.chromium.org/10975/diff/1/4
File src/unicode.h (right):

http://codereview.chromium.org/10975/diff/1/4#newcode253
Line 253: static int Convert(uchar c,
I'm guessing that FoldCase should correspond to the Canonicalize function of
ECMAScript. Is this correct?

http://codereview.chromium.org/10975/diff/1/4#newcode259
Line 259: static int Convert(uchar c,
And UnfoldCase is the inverse of FoldCase, which can return more than one
character, since FoldCase isn't injective.

http://codereview.chromium.org/10975/diff/1/5
File test/cctest/test-regexp.cc (right):

http://codereview.chromium.org/10975/diff/1/5#newcode735
Line 735: CHECK_EQ(0x212A, unfolded[2]);
Is UnfoldCase supposed to be the inverse of the Canonicalize function in
ECMAScript? 
If it is, then the Kelvin sign should not be included. You cannot Canonicalize
yourself into ASCII from something outside of ASCII (Canonicalize step 5).

http://codereview.chromium.org/10975/diff/1/5#newcode738
Line 738: // Long s
do.

[Christian Plesner Hansen, 2008-11-17 07:48:57]

http://codereview.chromium.org/10975/diff/1/4
File src/unicode.h (right):

http://codereview.chromium.org/10975/diff/1/4#newcode253
Line 253: static int Convert(uchar c,
No, this is case folding exactly as it is specified in the unicode spec.  We
will implement canonicalize on top of it.

http://codereview.chromium.org/10975/diff/1/4#newcode259
Line 259: static int Convert(uchar c,
It's not a true inverse, it's gives the equivalence classes defined by the fold
mapping.  Given a character x it yields the set of characters that fold to the
same as x.  A true inverse would map 'A' to {} since nothing folds to 'A'.

http://codereview.chromium.org/10975/diff/1/5
File test/cctest/test-regexp.cc (right):

http://codereview.chromium.org/10975/diff/1/5#newcode735
Line 735: CHECK_EQ(0x212A, unfolded[2]);
As mentioned, FoldCase is not meant to be Canonicalize.  I expect to implement
the special case for ASCII in v8 itself.

[Lasse Reichstein, 2008-11-17 08:14:35]

http://codereview.chromium.org/10975/diff/1/4
File src/unicode.h (right):

http://codereview.chromium.org/10975/diff/1/4#newcode253
Line 253: static int Convert(uchar c,
Ok. Have we checked this always give the correct result? 
If I understand it correctly, the Unicode folding algorithm converts to
lower-case (with extra complications like context sensitivity, and equivalence
classes that can contain several upper-case letters), whereas Canonicalize
converts all characters to a single character, upper-case if possible, otherwise
the character itself.

If there are two upper-case characters that fold to the same lower-case
character (i.e., are in the same equivalence class), they would be equivalent
modulo folding, but at most one of the upper-case characters can be "the
upper-case equivalent" of the lower-case character wrt. Canonicalize.

[Christian Plesner Hansen, 2008-11-17 08:26:21]

http://codereview.chromium.org/10975/diff/1/4
File src/unicode.h (right):

http://codereview.chromium.org/10975/diff/1/4#newcode253
Line 253: static int Convert(uchar c,
Unicode defines several mappings, including a 1:1 mapping, a 1:n mapping and a
locale-sensitive 1:n mapping.  FoldCase implements the simple 1:1 mapping.

Anyway, you're right: we can't actually use case folding to implement
canonicalize.  We need ToUpperCase (which we already have) and its equivalence
class mapping instead.  I should have guessed that the spec would use it's own
crazy magic kind of case independence rather than the one in the unicode
standard.

[Christian Plesner Hansen, 2008-11-17 09:35:39]

I have uploaded a new version of the code that implements canonicalize as
specified in ECMA-262 and an inverse, uncanonicalize.

[Lasse Reichstein, 2008-11-17 10:30:42]

LGTM!
We are not officially bug-compatible with the standard :)

http://codereview.chromium.org/10975/diff/802/606
File test/cctest/test-regexp.cc (right):

http://codereview.chromium.org/10975/diff/802/606#newcode725
Line 725: char upper = lower + ('A' - 'a');
Would it be reasonable to check every code point in the range 0..0xffff, to see
that it agrees with toUpperCase?

[Christian Plesner Hansen, 2008-11-17 10:47:33]

http://codereview.chromium.org/10975/diff/802/606
File test/cctest/test-regexp.cc (right):

http://codereview.chromium.org/10975/diff/802/606#newcode725
Line 725: char upper = lower + ('A' - 'a');
Certainly would; I've added it.