third_party/WebKit/LayoutTests/fast/encoding/char-decoding-invalid-trail.html - Issue 1721373002: UTF-8 detector for pages missing encoding info

Side by Side Diff: third_party/WebKit/LayoutTests/fast/encoding/char-decoding-invalid-trail.html

Issue 1721373002: UTF-8 detector for pages missing encoding info (Closed) Base URL: https://chromium.googlesource.com/chromium/src.git@master

Patch Set: Created 4 years, 9 months ago

Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.

Jump to:

View unified diff | Download patch

« third_party/WebKit/LayoutTests/accessibility/element-role-mapping-normal-expected.txt ('K') | « third_party/WebKit/LayoutTests/accessibility/element-role-mapping-normal-expected.txt ('k') | third_party/WebKit/LayoutTests/fast/encoding/char-decoding-invalid-trail-expected.txt » ('j') | third_party/WebKit/LayoutTests/fast/encoding/css-charset-evil/css-charset-evil-a1.css » ('J')
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Hide Comments ('s')

OLD	NEW
1 <!DOCTYPE html>	1 <!DOCTYPE html>

2 <script src="../../resources/js-test.js"></script>	2 <script src="../../resources/js-test.js"></script>

3 <script src="resources/char-decoding-utils.js"></script>	3 <script src="resources/char-decoding-utils.js"></script>

4 <script>	4 <script>

5	5

6 description("Test encoding behavior for sequences with invalid trail bytes");	6 description("Test encoding behavior for sequences with invalid trail bytes");

7	7

8 // UTF-8 codec emits replacement characters	8 // UTF-8 codec emits replacement characters

9 testDecode('UTF-8', '%C3%22', 'U+FFFD/U+0022');	9 testDecode('UTF-8', '%C3%22', 'U+FFFD/U+0022');

10 testDecode('UTF-8', '%E2%22', 'U+FFFD/U+0022');	10 testDecode('UTF-8', '%E2%22', 'U+FFFD/U+0022');

11 testDecode('UTF-8', '%E2%A0%22', 'U+FFFD/U+FFFD/U+0022');	11 testDecode('UTF-8', '%E2%A0%22', 'U+FFFD/U+FFFD/U+0022');

12 testDecode('UTF-8', '%F0%90%80%22', 'U+FFFD/U+FFFD/U+FFFD/U+0022');	12 testDecode('UTF-8', '%F0%90%80%22', 'U+FFFD/U+FFFD/U+FFFD/U+0022');

13	13

14 // When a trail byte is in the ASCII range and lead+trail does not	14 // When a trail byte is in the ASCII range and lead+trail does not

15 // encode a character, the ASCII trail byte has to be added back to	15 // encode a character, the ASCII trail byte has to be added back to

16 // the stream. Otherwise, lead+trail as a whole is turned to U+FFFD.	16 // the stream. Otherwise, lead+trail as a whole is turned to U+FFFD.

17 testDecode('EUC-KR', '%C4%22', 'U+FFFD/U+0022');	17 testDecode('EUC-KR', '%C4%22', 'U+FFFD/U+0022');

18 testDecode('EUC-KR', '%C4%5C', 'U+FFFD/U+005C');	18 testDecode('EUC-KR', '%C4%5C', 'U+FFFD/U+005C');

19 testDecode('EUC-KR', '%C4%7B', 'U+FFFD/U+007B');	19 testDecode('EUC-KR', '%C4%7B', 'U+FFFD/U+007B');

20 testDecode('EUC-KR', '%C6%53', 'U+FFFD/U+0053');	20 testDecode('EUC-KR', '%C6%53', 'U+FFFD/U+0053');

21 testDecode('EUC-KR', '%C7%41', 'U+FFFD/U+0041');	21 testDecode('EUC-KR', '%C7%41', 'U+FFFD/U+0041');

22 testDecode('EUC-KR', '%C7%81', 'U+FFFD');
Jinsuk Kim 2016/03/07 07:36:37 Remove this test since %C7%81 happens to be a vali Remove this test since %C7%81 happens to be a valid Unicode sequence (U+01C1). This CL identifies the test document as UTF-8 document and correctly decodes the character as such, not U+FFFD any more. jungshik at Google 2016/03/24 06:15:08 Wait. This test should NOT be affected by this CL Show quoted text On 2016/03/07 07:36:37, Jinsuk wrote: > Remove this test since %C7%81 happens to be a valid Unicode sequence (U+01C1). > This CL identifies the test document as UTF-8 document and correctly decodes the > character as such, not U+FFFD any more. Wait. This test should NOT be affected by this CL. If it is affected, there's something wrong. This test wants to make sure that '\xC7 \x81' is 'decoded' to U+FFFD when the encoding is explicitly set to EUC-KR. OTOH, this CL is supposed to affect pages WITHOUT any specified. Jinsuk Kim 2016/03/25 02:15:42 You're right - there was a bug in XMLHtttpRequest Show quoted text On 2016/03/24 06:15:08, jshin (jungshik at google) wrote: > On 2016/03/07 07:36:37, Jinsuk wrote: > > Remove this test since %C7%81 happens to be a valid Unicode sequence (U+01C1). > > This CL identifies the test document as UTF-8 document and correctly decodes > the > > character as such, not U+FFFD any more. > > Wait. This test should NOT be affected by this CL. If it is affected, there's > something wrong. > This test wants to make sure that '\xC7 \x81' is 'decoded' to U+FFFD when the > encoding is explicitly set to EUC-KR. > > OTOH, this CL is supposed to affect pages WITHOUT any specified. You're right - there was a bug in XMLHtttpRequest passing text encoding (EUC-KR) only, NOT the encoding source (EncodingFromHTTPheader) when creating TextResourceDecoder instance. This was putting UTF-8 encoding detector into action which override the text encoding when the detection returns true. Reverted the test file and fixed the bug by setting the encoding source as well. Please see XMLHttpRequest.cpp.
23 testDecode('EUC-KR', '%FE%A1', 'U+FFFD');	22 testDecode('EUC-KR', '%FE%A1', 'U+FFFD');

24	23

25 // TODO(jshin): Add more tests for EUC-JP.	24 // TODO(jshin): Add more tests for EUC-JP.

26 testDecode('EUC-JP', '%8F%A1%A1', 'U+FFFD');	25 testDecode('EUC-JP', '%8F%A1%A1', 'U+FFFD');

27 testDecode('EUC-JP', '%8F%A1%81%22', 'U+FFFD/U+FFFD/U+0022');	26 testDecode('EUC-JP', '%8F%A1%81%22', 'U+FFFD/U+FFFD/U+0022');

28 testDecode('EUC-JP', '%8F%A1%22', 'U+FFFD/U+FFFD/U+0022');	27 testDecode('EUC-JP', '%8F%A1%22', 'U+FFFD/U+FFFD/U+0022');

29 testDecode('EUC-JP', '%8E%8E%A1', 'U+FFFD/U+FF61');	28 testDecode('EUC-JP', '%8E%8E%A1', 'U+FFFD/U+FF61');

30 testDecode('EUC-JP', '%8E%E0', 'U+FFFD/U+FFFD');	29 testDecode('EUC-JP', '%8E%E0', 'U+FFFD/U+FFFD');

31	30

32 testDecode('Big5', '%A1%22', 'U+FFFD/U+0022');	31 testDecode('Big5', '%A1%22', 'U+FFFD/U+0022');

(...skipping 14 matching lines...) Expand all Loading...
47 testDecode('Shift_JIS', '%85%7B', 'U+FFFD/U+007B');	46 testDecode('Shift_JIS', '%85%7B', 'U+FFFD/U+007B');

48 testDecode('Shift_JIS', '%87%7B', 'U+FFFD/U+007B');	47 testDecode('Shift_JIS', '%87%7B', 'U+FFFD/U+007B');

49 testDecode('Shift_JIS', '%98%7E', 'U+FFFD/U+007E');	48 testDecode('Shift_JIS', '%98%7E', 'U+FFFD/U+007E');

50 testDecode('Shift_JIS', '%FC%5B', 'U+FFFD/U+005B');	49 testDecode('Shift_JIS', '%FC%5B', 'U+FFFD/U+005B');

51	50

52 // See https://www.w3.org/Bugs/Public/show_bug.cgi?id=28141#c4	51 // See https://www.w3.org/Bugs/Public/show_bug.cgi?id=28141#c4

53 // We're different from the encoding spec as of 2015-03-18.	52 // We're different from the encoding spec as of 2015-03-18.

54 testDecode('shift_jis', '%EB%9F', 'U+FFFD');	53 testDecode('shift_jis', '%EB%9F', 'U+FFFD');

55	54

56 </script>	55 </script>

OLD	NEW