Pass more hints to encoding detector.

Pass more hints to encoding detector.

http://chart.fisco.co.jp/fisco/cgi-bin/index.cgi was incorrectly detected as GBK
though it should be EUC-JP. Passing URL or user language "ja" fixes this issue.

* Add |hintUrl| and |hintUserLanguage| arguments to detectTextEncoding().

* TextResourceDecoder passes defaultLanguage() to detectTextEncoding().

* TextResourceDecoderBuilder passes URL information to TextResourceDecoder, and
  TextResourceDecoder passes it to detectTextEncoding().

BUG=682978
Review-Url: https://codereview.chromium.org/2648703003
Cr-Commit-Position: refs/heads/master@{#445317}
Committed: https://chromium.googlesource.com/chromium/src/+/269b6bc66eb96f69d61d814ea7ec1ffef695eee3

[tkent, 2017-01-20 04:57:42]

The CQ bit was checked by tkent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-20 04:57:51]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[tkent, 2017-01-20 05:04:19]

The CQ bit was checked by tkent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-20 05:04:44]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[tkent, 2017-01-20 05:05:06]

Patchset #1 (id:1) has been deleted

[tkent, 2017-01-20 05:07:36]

Description was changed from

==========
Pass more hints to encoding detector.

http://chart.fisco.co.jp/fisco/cgi-bin/index.cgi was incorrectly detected as GBK
though it should be EUC-JP. Passing URL or user language "ja" fixes this issue.

BUG=682978
==========

to

==========
Pass more hints to encoding detector.

http://chart.fisco.co.jp/fisco/cgi-bin/index.cgi was incorrectly detected as GBK
though it should be EUC-JP. Passing URL or user language "ja" fixes this issue.

* Add |hintUrl| and |hintUserLanguage| arguments to detectTextEncoding().

* TextResourceDecoder passes defaultLanguage() to detectTextEncoding().

* TextResourceDecoderBuilder passes URL information to TextResourceDecoder.

BUG=682978
==========

[tkent, 2017-01-20 05:21:28]

The CQ bit was checked by tkent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-20 05:21:36]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[tkent, 2017-01-20 05:21:59]

Patchset #1 (id:20001) has been deleted

[commit-bot: I haz the power, 2017-01-20 05:59:57]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-20 05:59:58]

Dry run: Try jobs failed on following builders:
  linux_android_rel_ng on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_androi...)
  cast_shell_linux on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/cast_shell_linu...)

[tkent, 2017-01-20 06:32:59]

The CQ bit was checked by tkent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-20 06:33:15]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[tkent, 2017-01-20 06:33:32]

Patchset #1 (id:40001) has been deleted

[tkent, 2017-01-20 06:47:08]

Description was changed from

==========
Pass more hints to encoding detector.

http://chart.fisco.co.jp/fisco/cgi-bin/index.cgi was incorrectly detected as GBK
though it should be EUC-JP. Passing URL or user language "ja" fixes this issue.

* Add |hintUrl| and |hintUserLanguage| arguments to detectTextEncoding().

* TextResourceDecoder passes defaultLanguage() to detectTextEncoding().

* TextResourceDecoderBuilder passes URL information to TextResourceDecoder.

BUG=682978
==========

to

==========
Pass more hints to encoding detector.

http://chart.fisco.co.jp/fisco/cgi-bin/index.cgi was incorrectly detected as GBK
though it should be EUC-JP. Passing URL or user language "ja" fixes this issue.

* Add |hintUrl| and |hintUserLanguage| arguments to detectTextEncoding().

* TextResourceDecoder passes defaultLanguage() to detectTextEncoding().

* TextResourceDecoderBuilder passes URL information to TextResourceDecoder, and
  TextResourceDecoder passes it to detectTextEncoding().

BUG=682978
==========

[commit-bot: I haz the power, 2017-01-20 07:15:53]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-20 07:15:54]

Dry run: Try jobs failed on following builders:
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[tkent, 2017-01-20 07:52:31]

The CQ bit was checked by tkent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-20 07:52:45]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[tkent, 2017-01-20 07:52:52]

Patchset #1 (id:60001) has been deleted

[tkent, 2017-01-20 07:58:02]

tkent@chromium.org changed reviewers:
+ jinsukkim@chromium.org

[tkent, 2017-01-20 07:58:03]

jinsukkim@, would you review this please?

[commit-bot: I haz the power, 2017-01-20 09:21:35]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-20 09:21:36]

Dry run: This issue passed the CQ dry run.

[Jinsuk Kim, 2017-01-20 21:50:47]

Thanks a lot for taking care of this!

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
File third_party/WebKit/Source/platform/text/TextEncodingDetector.cpp (right):

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
third_party/WebKit/Source/platform/text/TextEncodingDetector.cpp:45: Language
language = UNKNOWN_LANGUAGE;
This is not necessary since LanguageFromCode handles it.

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
File third_party/WebKit/Source/platform/text/TextEncodingDetector.h (right):

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
third_party/WebKit/Source/platform/text/TextEncodingDetector.h:53: const
WTF::String& hintUrl,
Can this be of type const char* like other parameters unless there is a
particular reason?

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
File third_party/WebKit/Source/platform/text/TextEncodingDetectorTest.cpp
(right):

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
third_party/WebKit/Source/platform/text/TextEncodingDetectorTest.cpp:36:
TEST(TextEncodingDetectorTest, UrlHintHelpsEUCJP) {
Can you add another EXPECT to show that additional hints help improve the
detection accuracy? Like detectTextEncoding without hints returning an encoding
which is not expected - like GBK as reported in the bug.

[tkent, 2017-01-23 00:02:24]

The CQ bit was checked by tkent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-23 00:02:36]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[tkent, 2017-01-23 00:02:57]

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
File third_party/WebKit/Source/platform/text/TextEncodingDetector.cpp (right):

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
third_party/WebKit/Source/platform/text/TextEncodingDetector.cpp:45: Language
language = UNKNOWN_LANGUAGE;
On 2017/01/20 at 21:50:47, Jinsuk Kim wrote:
> This is not necessary since LanguageFromCode handles it.

Done.

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
File third_party/WebKit/Source/platform/text/TextEncodingDetector.h (right):

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
third_party/WebKit/Source/platform/text/TextEncodingDetector.h:53: const
WTF::String& hintUrl,
On 2017/01/20 at 21:50:47, Jinsuk Kim wrote:
> Can this be of type const char* like other parameters unless there is a
particular reason?

Done.

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
File third_party/WebKit/Source/platform/text/TextEncodingDetectorTest.cpp
(right):

https://codereview.chromium.org/2648703003/diff/80001/third_party/WebKit/Sour...
third_party/WebKit/Source/platform/text/TextEncodingDetectorTest.cpp:36:
TEST(TextEncodingDetectorTest, UrlHintHelpsEUCJP) {
On 2017/01/20 at 21:50:47, Jinsuk Kim wrote:
> Can you add another EXPECT to show that additional hints help improve the
detection accuracy? Like detectTextEncoding without hints returning an encoding
which is not expected - like GBK as reported in the bug.

Done.

[Jinsuk Kim, 2017-01-23 00:08:23]

Thanks! lgtm.

[commit-bot: I haz the power, 2017-01-23 02:49:06]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-23 02:49:07]

Dry run: This issue passed the CQ dry run.

[tkent, 2017-01-23 03:15:40]

The CQ bit was checked by tkent@chromium.org

[commit-bot: I haz the power, 2017-01-23 03:15:53]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-23 03:19:58]

CQ is committing da patch.

Bot data: {"patchset_id": 100001, "attempt_start_ts": 1485141340873560,
"parent_rev": "e4067a31f79f739d9b4a64759ba18864ced18eac", "commit_rev":
"269b6bc66eb96f69d61d814ea7ec1ffef695eee3"}

[commit-bot: I haz the power, 2017-01-23 03:20:29]

Description was changed from

==========
Pass more hints to encoding detector.

http://chart.fisco.co.jp/fisco/cgi-bin/index.cgi was incorrectly detected as GBK
though it should be EUC-JP. Passing URL or user language "ja" fixes this issue.

* Add |hintUrl| and |hintUserLanguage| arguments to detectTextEncoding().

* TextResourceDecoder passes defaultLanguage() to detectTextEncoding().

* TextResourceDecoderBuilder passes URL information to TextResourceDecoder, and
  TextResourceDecoder passes it to detectTextEncoding().

BUG=682978
==========

to

==========
Pass more hints to encoding detector.

http://chart.fisco.co.jp/fisco/cgi-bin/index.cgi was incorrectly detected as GBK
though it should be EUC-JP. Passing URL or user language "ja" fixes this issue.

* Add |hintUrl| and |hintUserLanguage| arguments to detectTextEncoding().

* TextResourceDecoder passes defaultLanguage() to detectTextEncoding().

* TextResourceDecoderBuilder passes URL information to TextResourceDecoder, and
  TextResourceDecoder passes it to detectTextEncoding().

BUG=682978

Review-Url: https://codereview.chromium.org/2648703003
Cr-Commit-Position: refs/heads/master@{#445317}
Committed:
https://chromium.googlesource.com/chromium/src/+/269b6bc66eb96f69d61d814ea7ec...
==========

[commit-bot: I haz the power, 2017-01-23 03:20:30]

Committed patchset #2 (id:100001) as
https://chromium.googlesource.com/chromium/src/+/269b6bc66eb96f69d61d814ea7ec...