Change the system font fallback to take lang attributes into account on Windows

Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

All test rebaselines are where lang attributes are used, and new
results use the correct font for the specified lang.

BUG=579492
Committed: https://crrev.com/9ab3e68a0f7b8b71ccca2257d85a06277cac73ff
Cr-Commit-Position: refs/heads/master@{#370703}

[kojii, 2016-01-20 09:19:07]

Description was changed from

==========
font-fallback-win

BUG=
==========

to

==========
font-fallback-win

BUG=
==========

[kojii, 2016-01-20 14:29:47]

Description was changed from

==========
font-fallback-win

BUG=
==========

to

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters for Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

BUG=579492
==========

[kojii, 2016-01-20 14:30:59]

Description was changed from

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters for Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

BUG=579492
==========

to

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

BUG=579492
==========

[kojii, 2016-01-20 18:20:38]

kojii@chromium.org changed reviewers:
+ drott@chromium.org, eae@chromium.org

[kojii, 2016-01-20 18:20:39]

PTAL.

[kojii, 2016-01-20 18:22:15]

Description was changed from

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

BUG=579492
==========

to

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

All test rebaselines are where lang attributes are used, and new
results use the correct font for the specified lang.

BUG=579492
==========

[drott, 2016-01-20 20:32:49]

The code changes look good to me. 

I see that a couple of Windows baselines are required, probably due to new font
choices. Would it be possible to highlight those and add a
inspector-protocol/layout-fonts/* test illustrating the improvements? The test
can be disabled on non-Windows platforms for now, but it would be useful to have
a baseline in the layout test results that shows which optimal font choices
should be made for which lang and lang+subtag combinations and get a failing
test result when we start to deviate from those. (And out of curiosity, it would
be good to see how the font choices improved before and after.)

[kojii, 2016-01-21 02:22:25]

PTAL.

On 2016/01/20 20:32:49, drott wrote:
> The code changes look good to me. 
> 
> I see that a couple of Windows baselines are required, probably due to new
font
> choices. Would it be possible to highlight those and add a
> inspector-protocol/layout-fonts/* test illustrating the improvements? The test
> can be disabled on non-Windows platforms for now, but it would be useful to
have
> a baseline in the layout test results that shows which optimal font choices
> should be made for which lang and lang+subtag combinations and get a failing
> test result when we start to deviate from those. (And out of curiosity, it
would
> be good to see how the font choices improved before and after.)

Actually your:
inspector-protocol/layout-fonts/languages-emoji-rare-glyphs.html
does that for us; I added a manual rebaseline result from the bot.

When lang="ja", using "SimSun" (a Chinese font) is quite bad. Mixing it with "MS
PGothic" in a paragraph is even worse. This patch fixes the result.

[drott, 2016-01-21 08:47:36]

On 2016/01/21 02:22:25, kojii wrote:
> PTAL.
> 
> On 2016/01/20 20:32:49, drott wrote:
> > The code changes look good to me. 
> > 
> > I see that a couple of Windows baselines are required, probably due to new
> font
> > choices. Would it be possible to highlight those and add a
> > inspector-protocol/layout-fonts/* test illustrating the improvements? The
test
> > can be disabled on non-Windows platforms for now, but it would be useful to
> have
> > a baseline in the layout test results that shows which optimal font choices
> > should be made for which lang and lang+subtag combinations and get a failing
> > test result when we start to deviate from those. (And out of curiosity, it
> would
> > be good to see how the font choices improved before and after.)
> 
> Actually your:
> inspector-protocol/layout-fonts/languages-emoji-rare-glyphs.html
> does that for us; I added a manual rebaseline result from the bot.
> 
> When lang="ja", using "SimSun" (a Chinese font) is quite bad. Mixing it with
"MS
> PGothic" in a paragraph is even worse. This patch fixes the result.

Yes, that shows one nice improvement, thanks for adding the rebaseline and
explanations. But this test only features "ja" and "zh_Hans" as language
attributes. I thought it would be good to fix a test baseline with more examples
and variety of lang attributes, illustrating how the new code resolves ambiguity
for Chinese fallback, traditional vs. simplified, and how it improves fallback
by observing the subtags after en_ that are shown in FontFallbackWinTest.cpp.
This would give us a better alarm signal if our fallback behavior degrades in
the future and gives us code coverage for the Fallback code on Windows.

[kojii, 2016-01-21 09:20:14]

PTAL.

On 2016/01/21 08:47:36, drott wrote:
> > 
> > Actually your:
> > inspector-protocol/layout-fonts/languages-emoji-rare-glyphs.html
> > does that for us; I added a manual rebaseline result from the bot.
> > 
> > When lang="ja", using "SimSun" (a Chinese font) is quite bad. Mixing it with
> "MS
> > PGothic" in a paragraph is even worse. This patch fixes the result.
> 
> Yes, that shows one nice improvement, thanks for adding the rebaseline and
> explanations. But this test only features "ja" and "zh_Hans" as language
> attributes. I thought it would be good to fix a test baseline with more
examples
> and variety of lang attributes, illustrating how the new code resolves
ambiguity
> for Chinese fallback, traditional vs. simplified, and how it improves fallback
> by observing the subtags after en_ that are shown in FontFallbackWinTest.cpp.
> This would give us a better alarm signal if our fallback behavior degrades in
> the future and gives us code coverage for the Fallback code on Windows.

Done, added a new test and its local result for your review. The test copies
javascript from emoji test, and text from
fast\text\international\lang-glyph-cache-separation.html
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit...
These 3 glyphs are commonly known "a shared code point where people wants to see
in proper font."

[drott, 2016-01-21 09:25:50]

> Done, added a new test and its local result for your review. The test copies
> javascript from emoji test, and text from
> fast\text\international\lang-glyph-cache-separation.html
>
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit...
> These 3 glyphs are commonly known "a shared code point where people wants to
see
> in proper font."

Excellent, that's exactly what I meant. Thanks, LGTM.

[kojii, 2016-01-21 09:29:39]

The CQ bit was checked by kojii@chromium.org

[commit-bot: I haz the power, 2016-01-21 09:30:09]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1607943004/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1607943004/60001

[commit-bot: I haz the power, 2016-01-21 10:20:18]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-01-21 10:20:19]

Try jobs failed on following builders:
  mac_chromium_rel_ng on tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[kojii, 2016-01-21 15:15:58]

The CQ bit was checked by kojii@chromium.org

[kojii, 2016-01-21 15:15:59]

The patchset sent to the CQ was uploaded after l-g-t-m from drott@chromium.org
Link to the patchset: https://codereview.chromium.org/1607943004/#ps80001
(title: "Fix TestExpectations for the new test")

[commit-bot: I haz the power, 2016-01-21 15:16:43]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1607943004/80001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1607943004/80001

[commit-bot: I haz the power, 2016-01-21 15:48:46]

Description was changed from

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

All test rebaselines are where lang attributes are used, and new
results use the correct font for the specified lang.

BUG=579492
==========

to

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

All test rebaselines are where lang attributes are used, and new
results use the correct font for the specified lang.

BUG=579492
==========

[commit-bot: I haz the power, 2016-01-21 15:48:47]

Committed patchset #5 (id:80001)

[commit-bot: I haz the power, 2016-01-21 15:50:29]

Description was changed from

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

All test rebaselines are where lang attributes are used, and new
results use the correct font for the specified lang.

BUG=579492
==========

to

==========
Change the system font fallback to take lang attributes into account on Windows

This patch changes the system font fallback logic on Windows to take
lang attributes into account.

When a code point is missing in all specified fonts, Blink determines
fonts to use by using Unicode properties for the character.

The UI locale is also used for characters of Asian scripts where
multiple scripts share a large set of the code points.

This patch takes the specified lang attribute to take precedence over
the UI locale. When language tag does not help to determine the script
for the unified ideographs, subtags such as script or country are also
used.

All test rebaselines are where lang attributes are used, and new
results use the correct font for the specified lang.

BUG=579492

Committed: https://crrev.com/9ab3e68a0f7b8b71ccca2257d85a06277cac73ff
Cr-Commit-Position: refs/heads/master@{#370703}
==========

[commit-bot: I haz the power, 2016-01-21 15:50:30]

Patchset 5 (id:??) landed as
https://crrev.com/9ab3e68a0f7b8b71ccca2257d85a06277cac73ff
Cr-Commit-Position: refs/heads/master@{#370703}