Convert Encoding API tests to W3C testharness.js

Convert Encoding API tests to W3C testharness.js

These tests should be upstreamed to W3C web-platform-tests.

BUG=368310, 368904
Committed: https://src.chromium.org/viewvc/blink?view=rev&revision=173047

[jsbell, 2014-04-25 22:43:47]

kbr@, jshin@ - please take a look?

Once we pass these we can upstream to the w3c web-platform-tests repository.

Note that the failures are identical, just a lot noisier.

[Ken Russell (switch to Gerrit), 2014-04-26 00:18:43]

LGTM, though I defer to jshin's review.

Would be good to add a BUG= to the CL.

[jungshik at Google, 2014-04-29 16:51:04]

Thank you for the CL and sorry for the delay. Would you consider my comments
below ?

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
File LayoutTests/fast/encoding/api/basics.html (right):

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/basics.html:34:
'z\xA2\u6C34\uD834\uDD1E\uDBFF\uDFFD', // z, cent, CJK water, G-Clef,
Private-use character
Shouldn't this sample include a BMP PUA code point in addition to a non-BMP PUA
code point? 

How about non-character code points? They're just passed through per spec, but
having them checked would be a good idea. 

How about overlong UTF-8 sequences (e.g. < U+0080 represented in 2-byte)? They'd
be turned to U+FFFD, right? Oh, perhaps, fatal-flags.html would be a better
place for overlong sequences. I mentioned it there.

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
File LayoutTests/fast/encoding/api/fatal-flag.html (right):

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/fatal-flag.html:17: { encoding: 'utf-8', input:
[0xE0, 0x80, 0xC0], name: 'invalid trail' },
Please, add overlong sequences here. 
<= U+007F  represented in 2 or more bytes (up to 6)
U+0080 ... U+07FF : represented in 3 or more bytes
U+0800 ... U+FFFF (other than surrogates) : represented in 4 or more bytes
U+FFFF .. U+10FFFF : represetned in 5 or more bytes

Also, add surrogate code points (paired or unpaired).  represented in 3-byte
sequences. A pair of surrogate code points each of them represented in 3-byte
sequence is not UTF-8 but CES-UTF8 (as used in Oracle DB).

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/fatal-flag.html:18: { encoding: 'utf-8', input:
[0xFC, 0x80, 0x80, 0x80, 0x80, 0x80], name: '> 0x10FFFF' },
Pls, also add a 6-byte sequence (valid in the original UTF-8 but not vailid any
more) starting with 0xFE.

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
File LayoutTests/fast/encoding/api/surrogate-pairs.html (right):

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/surrogate-pairs.html:10: expected: [97, 98, 99,
49, 50, 51],
nit: Would you use hexadecimal here, too?

[jsbell, 2014-04-29 17:39:41]

On 2014/04/26 00:18:43, Ken Russell wrote:
> LGTM, though I defer to jshin's review.
> 
> Would be good to add a BUG= to the CL.

Done!

[jsbell, 2014-04-29 21:41:17]

Thanks for the feedback. Can you take another look?

As a follow-up CL I will try and consolidate the failures due to Blink's UTF-16
surrogate pair handling to one test.

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
File LayoutTests/fast/encoding/api/basics.html (right):

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/basics.html:34:
'z\xA2\u6C34\uD834\uDD1E\uDBFF\uDFFD', // z, cent, CJK water, G-Clef,
Private-use character
On 2014/04/29 16:51:04, Jungshik Shin wrote:
> Shouldn't this sample include a BMP PUA code point in addition to a non-BMP
PUA
> code point? 

Good idea. (I inherited this sequence from some other tests long ago.)

Done. U+F8FF

> How about non-character code points? They're just passed through per spec, but
> having them checked would be a good idea. 

Done. U+FFFE (byte-swapped BOM non-character)

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
File LayoutTests/fast/encoding/api/fatal-flag.html (right):

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/fatal-flag.html:17: { encoding: 'utf-8', input:
[0xE0, 0x80, 0xC0], name: 'invalid trail' },
On 2014/04/29 16:51:04, Jungshik Shin wrote:
> Please, add overlong sequences here. 
> <= U+007F  represented in 2 or more bytes (up to 6)
> U+0080 ... U+07FF : represented in 3 or more bytes
> U+0800 ... U+FFFF (other than surrogates) : represented in 4 or more bytes
> U+FFFF .. U+10FFFF : represetned in 5 or more bytes
> 
> Also, add surrogate code points (paired or unpaired).  represented in 3-byte
> sequences. A pair of surrogate code points each of them represented in 3-byte
> sequence is not UTF-8 but CES-UTF8 (as used in Oracle DB). 

Done.

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/fatal-flag.html:18: { encoding: 'utf-8', input:
[0xFC, 0x80, 0x80, 0x80, 0x80, 0x80], name: '> 0x10FFFF' },
On 2014/04/29 16:51:04, Jungshik Shin wrote:
> Pls, also add a 6-byte sequence (valid in the original UTF-8 but not vailid
any
> more) starting with 0xFE.  

Done.

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
File LayoutTests/fast/encoding/api/surrogate-pairs.html (right):

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/surrogate-pairs.html:10: expected: [97, 98, 99,
49, 50, 51],
On 2014/04/29 16:51:04, Jungshik Shin wrote:
> nit: Would you use hexadecimal here, too? 

Done.

https://codereview.chromium.org/240283013/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/surrogate-pairs.html:40: expected: [239, 191, 189,
239, 191, 189],
Here too.

[jsbell, 2014-04-29 23:52:12]

On 2014/04/29 21:41:17, jsbell wrote:
> As a follow-up CL I will try and consolidate the failures due to Blink's
UTF-16
> surrogate pair handling to one test.

Actually, I did that here in patch set #5.

The failures are:

ascii-supersets.html - iso-2022-jp and shift_jis code points, crbug.com/277062,
which jshin@ is handling
encoding-labels-expected.txt - big5-hkscs vs. big5, crbug.com/277040, not urgent
encoding-labels-expected.txt - replacement encodings, crbug.com/277037, waiting
on an L*G*T*M from a wtf OWNER
utf16-surrogates-expected.txt - blink doesn't reject bad UTF-16 surrogates at
all; fixing this may be a compat issue (?), and may require special-casing the
codec further when used from the API

[jungshik at Google, 2014-04-30 23:37:31]

Thank you for the update. LGTM

On 2014/04/29 23:52:12, jsbell wrote:
> On 2014/04/29 21:41:17, jsbell wrote:
> > As a follow-up CL I will try and consolidate the failures due to Blink's
> UTF-16
> > surrogate pair handling to one test.
> 
> Actually, I did that here in patch set #5.
> 
> The failures are:
> 
> ascii-supersets.html - iso-2022-jp and shift_jis code points,
crbug.com/277062,
> which jshin@ is handling

I'll deal with SJIS after the initial round of ICU 52 roll-in. As for
ISO-2022-JP, 
https://www.w3.org/Bugs/Public/show_bug.cgi?id=16685 has to be resolved in the
spec (especially RFC 1447 compliance vs FF/IE behavior). 


> encoding-labels-expected.txt - big5-hkscs vs. big5, crbug.com/277040, not
urgent
> encoding-labels-expected.txt - replacement encodings, crbug.com/277037,
waiting
> on an L*G*T*M from a wtf OWNER

> utf16-surrogates-expected.txt - blink doesn't reject bad UTF-16 surrogates at
> all; fixing this may be a compat issue (?), and may require special-casing the
> codec further when used from the API

Perhaps, we need a bug on this if none has been filed?

[jsbell, 2014-04-30 23:49:48]

On 2014/04/30 23:37:31, Jungshik Shin wrote:
> Thank you for the update. LGTM
> 
> On 2014/04/29 23:52:12, jsbell wrote:
> > On 2014/04/29 21:41:17, jsbell wrote:
> > > As a follow-up CL I will try and consolidate the failures due to Blink's
> > UTF-16
> > > surrogate pair handling to one test.
> > 
> > Actually, I did that here in patch set #5.
> > 
> > The failures are:
> > 
> > ascii-supersets.html - iso-2022-jp and shift_jis code points,
> crbug.com/277062,
> > which jshin@ is handling
> 
> I'll deal with SJIS after the initial round of ICU 52 roll-in. As for
> ISO-2022-JP, 
> https://www.w3.org/Bugs/Public/show_bug.cgi?id=16685 has to be resolved in the
> spec (especially RFC 1447 compliance vs FF/IE behavior). 
> 
> 
> > encoding-labels-expected.txt - big5-hkscs vs. big5, crbug.com/277040, not
> urgent
> > encoding-labels-expected.txt - replacement encodings, crbug.com/277037,
> waiting
> > on an L*G*T*M from a wtf OWNER
> 
> > utf16-surrogates-expected.txt - blink doesn't reject bad UTF-16 surrogates
at
> > all; fixing this may be a compat issue (?), and may require special-casing
the
> > codec further when used from the API
> 
> Perhaps, we need a bug on this if none has been filed?

Done: crbug.com/368904

[jsbell, 2014-04-30 23:50:11]

The CQ bit was checked by jsbell@chromium.org

[commit-bot: I haz the power, 2014-04-30 23:50:33]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/jsbell@chromium.org/240283013/70001

[commit-bot: I haz the power, 2014-05-01 00:56:54]

Change committed as 173047