Make all the single byte encodings compliant to the encoding spec.

Make all the single byte encodings compliant to the encoding spec.

1. Replace the current encoding alias list (heavily patched) with our own
HTML5-specific alias list. It's mostly generated from encoding.json, which
is in turn derived from the WHATWG Encoding living standard. The most notable
difference is that UTF-32 entries are kept until bug 417850 is resolved.
Two other differences are:

 a. Two aliases for iso-8859-8-i (logical and csiso88598i) are not listed.
 They're dealt with in Blink.

 b. Chinese (gb*, big5*) aliases are not yet aligned to the encoding
spec pending our decision on the unification of Big5 / Big5-HKSCS and
GBK / GB18030.

2. Replace all the single-byte mapping tables with what's automatically
generated with scripts/single-byte-gen.sh that uses index-* files downloaded
from the WHATWG spec site. This will fix the decoding (ToUnicode) of
windows-874 and windows-1253 while removing a lot of fallback/spurrious
mapping entries in encoding direction ('FromUnicode') in a number of encodings.

3. Regenerate the ICU binary data files for Linux/Mac/Android/Windows/CrOS.

4. Remove now obsolete noop-*ucm files used to make ISO-2022-CN* decoder
to turn an empty string. They're not necessary any more because ISO-2022-CN*
were made 'replacement' encodings in Blink and our version of ICU does not
have any code for ISO-2022-CN* any more.

This cuts down the data size by 15kB. On Android, there's virtually
no change in the data size because the previous data file on Android
accidentally had smaller locale data for nb and ms.

BUG=412053
TEST=browser_tests --gtest_filter="*ncoding*"
TEST=net_unittest --gtest_filter="*ilenameUtil*"
TEST=base_unittests --gtest_filter="*Conv*"
TEST=Blink: fast/encoding/*
TEST=http://www.w3.org/International/tests/repository/encoding/indexes/results-indexes
TEST=http://www.w3.org/International/tests/repository/encoding/indexes/results-aliases
TEST=http://www.w3.org/International/tests/repository/run?manifest=encoding/indexes&test=windows-1253_test
TEST=http://www.w3.org/International/tests/repository/run?manifest=encoding/indexes&test=windows-874_test
R=jsbell@chromium.org

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=292447

[jungshik at Google, 2014-10-03 23:56:43]

jshin@chromium.org changed reviewers:
+ jsbell@chromium.org

[jungshik at Google, 2014-10-03 23:56:44]

Can you take a look? Thanks. 

With this change, we're multiple steps closer to the WHATWG encoding spec.
Coming up next is Big5/Big5-HKSCS (I'm now convinced that we'd better merge them
into one) and EUC-KR/Windows-949.

[jsbell, 2014-10-04 00:15:26]

https://codereview.chromium.org/598383002/diff/20001/scripts/single_byte_gen.sh
File scripts/single_byte_gen.sh (right):

https://codereview.chromium.org/598383002/diff/20001/scripts/single_byte_gen....
scripts/single_byte_gen.sh:31: encodings="ibm866 iso-8859-2 iso-8859-3
iso-8859-4 iso-8859-5 iso-8859-6\
Consider including a note about iso-8859-8-i here too, since it's conspicuously
absent.

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
File source/data/mappings/convrtrs.txt (right):

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:120: UTR22           # # Name format specified
by http://www.unicode.org/unicode/reports/tr22/
Nit: double # ?

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:121: HTML            # WHATWG's encoding spec
Include URL?

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:143: ibm866-html
Does the first (non-indented) entry get treated as an alias as well? It looks
like it did in the old file.

[jungshik at Google, 2014-10-04 09:32:41]

https://codereview.chromium.org/598383002/diff/20001/scripts/single_byte_gen.sh
File scripts/single_byte_gen.sh (right):

https://codereview.chromium.org/598383002/diff/20001/scripts/single_byte_gen....
scripts/single_byte_gen.sh:31: encodings="ibm866 iso-8859-2 iso-8859-3
iso-8859-4 iso-8859-5 iso-8859-6\
On 2014/10/04 00:15:26, jsbell wrote:
> Consider including a note about iso-8859-8-i here too, since it's
conspicuously
> absent.
> 

Will do.

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
File source/data/mappings/convrtrs.txt (right):

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:120: UTR22           # # Name format specified
by http://www.unicode.org/unicode/reports/tr22/
On 2014/10/04 00:15:26, jsbell wrote:
> Nit: double # ?

Thanks. Will fix

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:121: HTML            # WHATWG's encoding spec
On 2014/10/04 00:15:26, jsbell wrote:
> Include URL?

Will add.

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:143: ibm866-html
On 2014/10/04 00:15:26, jsbell wrote:
> Does the first (non-indented) entry get treated as an alias as well? It looks
> like it did in the old file.

Yeah, it does. It's also used to refer to a ucm file name. Hmm.... we can drop
'-html' suffix, but the upstream may not like it when this change is upstreamed.

[jsbell, 2014-10-07 17:13:21]

lgtm; I sanity checked the aliases and some of the single-byte mappings.

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
File source/data/mappings/convrtrs.txt (right):

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:123: MIME            # Source:
http://www.iana.org/assignments/character-sets
Do we reference the different standards tags from Chromium? It looks like you're
removing the tags apart from HTML and MIME do we ever interact with these from
code and do we ever specify MIME? 

(I should learn more about ICU and how we use it.)

https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
source/data/mappings/convrtrs.txt:143: ibm866-html
On 2014/10/04 09:32:40, Jungshik Shin wrote:
> On 2014/10/04 00:15:26, jsbell wrote:
> > Does the first (non-indented) entry get treated as an alias as well? It
looks
> > like it did in the old file.
> 
> Yeah, it does. It's also used to refer to a ucm file name. Hmm.... we can drop
> '-html' suffix, but the upstream may not like it when this change is
upstreamed.
> 

We could reject the "-html" in Blink (can land later). Other ideas?

[jungshik at Google, 2014-10-13 22:38:12]

On 2014/10/07 17:13:21, jsbell wrote:
> lgtm; I sanity checked the aliases and some of the single-byte mappings.
> 
>
https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
> File source/data/mappings/convrtrs.txt (right):
> 
>
https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
> source/data/mappings/convrtrs.txt:123: MIME            # Source:
> http://www.iana.org/assignments/character-sets
> Do we reference the different standards tags from Chromium? It looks like
you're
> removing the tags apart from HTML and MIME do we ever interact with these from
> code and do we ever specify MIME? 
> 
> (I should learn more about ICU and how we use it.)
> 
>
https://codereview.chromium.org/598383002/diff/20001/source/data/mappings/con...
> source/data/mappings/convrtrs.txt:143: ibm866-html
> On 2014/10/04 09:32:40, Jungshik Shin wrote:
> > On 2014/10/04 00:15:26, jsbell wrote:
> > > Does the first (non-indented) entry get treated as an alias as well? It
> looks
> > > like it did in the old file.
> > 
> > Yeah, it does. It's also used to refer to a ucm file name. Hmm.... we can
drop
> > '-html' suffix, but the upstream may not like it when this change is
> upstreamed.
> > 
> 
> We could reject the "-html" in Blink (can land later). Other ideas?


I thought of a few different ways to handle this :

1) I thought what I wrote above about the ucm file name reference might not be
true. It turned out that I was right unfortunately. That is, listing
'windows-1258' (without '-html' suffix) in convrtrs.txt while listing the ucm
file name ('windows-1258-html') and setting the converter name inside the file
to 'windows-1258' does not work the way I expect it to work. The alias table
(convrtrs.txt) does need a reference to the ucm file name. 

2) Yeah, we can reject '-html' in blink. There are 3 variations:

  a. Tag all the aliases we want to include with 'HTML' tag in convrtrs.txt
except for those with '-html' suffix. And add only those with 'HTML' tag to
Blink's encoding table (using ICU's ucnv_openStandardNames(.... "HTML" ...) ) 
  b. Check whether an alias ends with '-html' and drop it (your suggestion)
  c. Tag those with '-html' suffix with 'EXCLUDE' (or something we made up) and
exclude it in Blink. 

It seems like 2b is the simplest (I'm hoping that it does not have much perf
impact).  2a requires adding several lines of code to the blink unless we can
use '#ifdef CHROME_ICU'.    

3) Drop '-html' suffix from the ucm file names. It'd be the simplests of all,
but the upstream may not like that. 

So, I'll go with 2b (that is no change in this CL and a couple of new lines in
Blink to reject '*-html'). 

Thank you for the review.

[jungshik at Google, 2014-10-13 23:07:08]

Committed patchset #4 (id:40001) manually as r292447 (presubmit successful).