icu local change part1

ICU 52 local changes part1

1. Remove all the obsolete patches. There are lots of them because most of
local patches to ICU 4.6.1 have either been accepted or become obsolete.
The largest local patch removed is our patches for CJ word breaker because
they were upstreamed.

Android didn't apply the CJK word breaker patch to ICU 4.6 to reduce the
data size. In a follow-up CL, we'll have an Android-specific change for this issue.

Besides, we don't include patches for files we locally add because the
patches for new files are redundant. Instead, they're mentioned in
README.chromium.

2. We don't need platform-specific headers any more (pmac, plinux, pwin, etc).
They're combined into a single file and all platforms we care about are
well-supported except for one issue on Android/QNX. putil.patch takes care
of it.


3. Breakiterator patches for a few remaining issues. We also use
a much smaller Khmer dictionary (upstream fix pending).

4. Converter
  - Introduced two WHATWG-encoding-standard-compliant mapping tables
    are added (derived directly from the spec with a script) for EUC-JP
    and CP866
  - Disabled various non-HTML5-encodings such as SCSU,BOCU, UTF-7, CESU-8
    saving ~30kB in the code size. Even though we link statically, they're
    still pulled in as a part of uconv.
  - Disabled ISO-2022-JP-[1-4] in ucnv2022.c
  - Removed a number of encoding alias entries in the alias table
    leading to ~40kB data size reduction.

5. Locale data : Haven't yet updated. We need to trim them substantially.

6. Unihan collation removal is now done with a script (scripts/remove_unihan.sh)

7. Updated timezone data to the latest (2014b) as of today.

8. Customized transliterator for Greek uppercasing

9. Updated data build related patches. The windows data build patch has yet
   to be updated.

10. The updated ICU data file/assembly source files are not included in this
    CL. They'll be updated in a separate CL.
    With all the size reduction changes applied, the data size went down
    from > 23MB to 12.4MB. However, it's still 2.5MB larger than ICU 4.6.1
    data. The locale data trimming will bring it down further.

11. Update README.chromium accordingly. The only exceptions are
item #5 and the android entry in item #3 (breakiterator. see #1 above)



BUG=132145, 76328
TEST=Following the procedure outlined in README.chromium, one can build
the icu data file.

R=jsbell@chromium.org, mark@chromium.org

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=262192

[jungshik at Google, 2014-04-04 09:29:52]

Mark, sorry to bother you with this gigantic CL. It's hard to say what to look
at here other than some sanity checking. Anyway, can you take a look? Thank you.


@jsbell, FYI, I'm adding EUC-JP and CP864 tables directly derived from the
algorithm and tables of the WHATWG encoding spec. I haven't updated Shift_JIS,
yet. I'll do that in a separate CL.

[jungshik at Google, 2014-04-04 18:04:03]

On 2014/04/04 09:29:52, Jungshik Shin wrote:
> Mark, sorry to bother you with this gigantic CL. It's hard to say what to look
> at here other than some sanity checking. Anyway, can you take a look? Thank
you.

Just going through README.chromium and files in scripts/ would be good enough, I
guess. 

> 
> @jsbell, FYI, I'm adding EUC-JP and CP864 tables directly derived from the
> algorithm and tables of the WHATWG encoding spec. I haven't updated Shift_JIS,
> yet. I'll do that in a separate CL.

The files relevant to that are in source/data/mappings/{euc-jp*,cp866*} and
scripts/*gen.sh

Thanks.

[jsbell, 2014-04-04 18:16:19]

crbug.com/277023 is about CP866, it could be added to the BUG= line. (The CL
description/changes mention 866, your comment mentions 864 - typo, or is there
something more subtle going on?)

https://codereview.chromium.org/224943002/diff/30001/icu52/scripts/eucjp_gen.sh
File icu52/scripts/eucjp_gen.sh (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/scripts/eucjp_gen....
icu52/scripts/eucjp_gen.sh:26: # *   Added the 34 decoding only (EUC-JP to
Unicode) entries from euc-jp-2007.ucm
Do we want to propose these as additions to the Encoding spec, if we believe
they're necessary for compat?

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/locm...
File icu52/source/common/locmap.c (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/locm...
icu52/source/common/locmap.c:32: #if 0
Comment on this?

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/stri...
File icu52/source/common/stringpiece.cpp (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/stri...
icu52/source/common/stringpiece.cpp:71: /* Microsft Visual Studios (even 2013)
complains about redefinition of this
Typos: "Microsoft Visual Studio" (+o -s)

(Or just use MSVC)

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/unic...
File icu52/source/common/unicode/uconfig.h (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/unic...
icu52/source/common/unicode/uconfig.h:272: #define
UCONFIG_NO_NON_HTML5_CONVERSION 1
It must be intentional, but having these two defines with names that only differ
by _ will be confusing.

(The patch is failing to download so I can't search easily to see where this is
set/used.)

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
File icu52/source/data/mappings/convrtrs.txt (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:331: gb18030 { IANA* }       ibm-1392 {
IBM* } windows-54936 { WINDOWS* } gb18030 { MIME* }
Maybe add a comment:

# CHROME: WHATWG Encoding Standard makes GBK an alias for GB18030
#         TODO (jshin): Decide if Chrome should follow spec. crbug.com/339862

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:448: # HTML5.
Nit: WHATWG Encoding Living Standard, not HTML5. (I don't know if W3C has
decided to start snapshotting it yet.)

But... it's probably better to use the same name everywhere in our ICU
patches/support. So I don't object to keeping it as HTML5 for simplicity.

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:560: ibm-1375_P100-2007 { UTR22* }   #
Big5-HKSCS-2004 with Unicode 3.1 mappings. This uses supplementary characters.
Maybe add comment:

# CHROME: WHATWG Encoding Standard has big5-hkscs as an alias for big5
#         TODO(jshin): Decide if Chrome should follow spec. crbug.com/277040

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:642: #CHROME : Make ISO-8859-9 an alias
to windows-1254 per HTML5. Move
Nit: Should probably be consistent with the comments, i.e. space after # and
consistent capitalization of the "CHROME:" prefix

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:671: # replacement encodings.
Could reference crbug.com/277037

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:682: # Chrome: Removed all EBCDIC code
pages.
This is awesome. We should let the TPMs know so they can blog it. :)

[Mark Mentovai, 2014-04-04 19:22:51]

I only looked closely at the scripts and README.chromium. I spot-checked parts
of the rest. Seems fine.

https://codereview.chromium.org/224943002/diff/30001/icu52/README.chromium
File icu52/README.chromium (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/README.chromium#ne...
icu52/README.chromium:44: See http://unicode.org/cldr/trac/ticket/3974
Nuke the tabs.

https://codereview.chromium.org/224943002/diff/30001/icu52/scripts/remove_uni...
File icu52/scripts/remove_unihan.sh (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/scripts/remove_uni...
icu52/scripts/remove_unihan.sh:6: cd `dirname $0`/../source/data/coll
Backticks suck, use $(…) instead.

And you should be quoting more judiciously in shell scripts. This line should be

cd "$(dirname "$0"/../source/data/coll)"

There are a few other things that could be quoted a little bit better in the
other scripts too.

[jungshik at Google, 2014-04-04 22:20:26]

Thanks a lot for the review. Updated the CL.

https://codereview.chromium.org/224943002/diff/30001/icu52/README.chromium
File icu52/README.chromium (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/README.chromium#ne...
icu52/README.chromium:44: See http://unicode.org/cldr/trac/ticket/3974
On 2014/04/04 19:22:52, Mark Mentovai wrote:
> Nuke the tabs.

Done.

https://codereview.chromium.org/224943002/diff/30001/icu52/scripts/eucjp_gen.sh
File icu52/scripts/eucjp_gen.sh (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/scripts/eucjp_gen....
icu52/scripts/eucjp_gen.sh:26: # *   Added the 34 decoding only (EUC-JP to
Unicode) entries from euc-jp-2007.ucm
On 2014/04/04 18:16:20, jsbell wrote:
> Do we want to propose these as additions to the Encoding spec, if we believe
> they're necessary for compat?
> 

I don't have a good sense of how important they're, but I was conservative in
keeping them (decoding only) for now. Anyway, I filed a bug (
https://www.w3.org/Bugs/Public/show_bug.cgi?id=25266 ).

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/locm...
File icu52/source/common/locmap.c (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/locm...
icu52/source/common/locmap.c:32: #if 0
On 2014/04/04 18:16:20, jsbell wrote:
> Comment on this?

Added a comment.

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/stri...
File icu52/source/common/stringpiece.cpp (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/stri...
icu52/source/common/stringpiece.cpp:71: /* Microsft Visual Studios (even 2013)
complains about redefinition of this
On 2014/04/04 18:16:20, jsbell wrote:
> Typos: "Microsoft Visual Studio" (+o -s)
> 
> (Or just use MSVC)
> 

Done.

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/unic...
File icu52/source/common/unicode/uconfig.h (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/common/unic...
icu52/source/common/unicode/uconfig.h:272: #define
UCONFIG_NO_NON_HTML5_CONVERSION 1
On 2014/04/04 18:16:20, jsbell wrote:
> It must be intentional, but having these two defines with names that only
differ
> by _ will be confusing.
> 
> (The patch is failing to download so I can't search easily to see where this
is
> set/used.)

A good catch. Thanks !! Actually, it's a typo. Moreover, I should have used
#ifndef instead of #if. 

More importantly, fixing the typo led me to find more places where I need to use
that check (linker complained about unresolved references). The size reduction
in libicuuc.so is 47kB (I think static linking does not help here and the size
reduction would be comparable in case of static linking).

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
File icu52/source/data/mappings/convrtrs.txt (right):

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:331: gb18030 { IANA* }       ibm-1392 {
IBM* } windows-54936 { WINDOWS* } gb18030 { MIME* }
On 2014/04/04 18:16:20, jsbell wrote:
> Maybe add a comment:
> 
> # CHROME: WHATWG Encoding Standard makes GBK an alias for GB18030
> #         TODO (jshin): Decide if Chrome should follow spec. crbug.com/339862

Thanks for the reminder. Done here as well as around GBK.

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:448: # HTML5.
On 2014/04/04 18:16:20, jsbell wrote:
> Nit: WHATWG Encoding Living Standard, not HTML5. (I don't know if W3C has
> decided to start snapshotting it yet.)
> 
> But... it's probably better to use the same name everywhere in our ICU
> patches/support. So I don't object to keeping it as HTML5 for simplicity.

Ok. I'll keep it simpler. :-)

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:560: ibm-1375_P100-2007 { UTR22* }   #
Big5-HKSCS-2004 with Unicode 3.1 mappings. This uses supplementary characters.
On 2014/04/04 18:16:20, jsbell wrote:
> Maybe add comment:
> 
> # CHROME: WHATWG Encoding Standard has big5-hkscs as an alias for big5
> #         TODO(jshin): Decide if Chrome should follow spec. crbug.com/277040

Done.

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:642: #CHROME : Make ISO-8859-9 an alias
to windows-1254 per HTML5. Move
On 2014/04/04 18:16:20, jsbell wrote:
> Nit: Should probably be consistent with the comments, i.e. space after # and
> consistent capitalization of the "CHROME:" prefix

Done.

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:671: # replacement encodings.
On 2014/04/04 18:16:20, jsbell wrote:
> Could reference crbug.com/277037

Done.

https://codereview.chromium.org/224943002/diff/30001/icu52/source/data/mappin...
icu52/source/data/mappings/convrtrs.txt:682: # Chrome: Removed all EBCDIC code
pages.
On 2014/04/04 18:16:20, jsbell wrote:
> This is awesome. We should let the TPMs know so they can blog it. :)

We never included EBCDIC converters before. What's removed is aliases for all
the EBCDIC converters. Well, it's still a couple of tens of kBs of savings.

[jungshik at Google, 2014-04-07 17:17:43]

Can you take another look? Thanks

[jsbell, 2014-04-07 17:45:52]

quick skim lgtm

(In the .sh files, the function look like they should be indented to me, but I
don't know if there's a convention there.)

[Mark Mentovai, 2014-04-07 17:51:22]

I agree about the indentation. LGTM otherwise.

[jungshik at Google, 2014-04-07 20:30:20]

Committed patchset #6 manually as r262192 (presubmit successful).