Use and add layout tests for BCP 47 language tag syntax validation.

Use and add layout tests for BCP 47 language tag syntax validation.

Media Source Extensions' TrackDefault() constructor and both the Web
Notification constructor and SWR.showNotification() function now validate
that the syntax of the language attribute conforms to the syntax described
in BCP 47.

This CL is part of a three-sided patch:
  [1] https://codereview.chromium.org/1115453003/
  [2] https://codereview.chromium.org/1110233002/
  [3] This CL.

BUG=442137

[Peter Beverloo, 2015-04-28 19:17:35]

peter@chromium.org changed reviewers:
+ philipj@opera.com

[Peter Beverloo, 2015-04-28 19:17:36]

+philipj since this touches MSE and the public/ API

[philipj_slow, 2015-04-29 13:56:05]

I'm a little bit skeptical about exposing BCP 47 validity to the Web, it seems
like more work than it's worth to reach interoperability here. In most contexts,
language strings are simply reflected as-is and if they are invalid they are
simply not used internally, but this fact isn't exposed to the Web.

Are there use cases for letting Web apps know that the language string wasn't
valid BCP 47, or is it just the fact that these specs say so?

[Peter Beverloo, 2015-04-29 13:59:29]

On 2015/04/29 13:56:05, philipj_UTC2 wrote:
> I'm a little bit skeptical about exposing BCP 47 validity to the Web, it seems
> like more work than it's worth to reach interoperability here. In most
contexts,
> language strings are simply reflected as-is and if they are invalid they are
> simply not used internally, but this fact isn't exposed to the Web.
> 
> Are there use cases for letting Web apps know that the language string wasn't
> valid BCP 47, or is it just the fact that these specs say so?

The latter - spec compatibility, both for MSE and Notifications.

This is just a very low priority thing that's been on a list for a long time, I
don't feel strongly in either way :-). The implementation work is trivial
enough.

There may well be use-cases, the i18n JavaScript API exposes and validates
similar functionality in the "locales" arguments.

[philipj_slow, 2015-04-29 14:00:06]

philipj@opera.com changed reviewers:
+ wolenetz@chromium.org

[philipj_slow, 2015-04-29 14:00:06]

wolenetz@, what are your thoughts on the MSE case? I have a feeling we've
discussed this at one point, but I can't find any open bugs. I understand that
MSE does what it does because the HTML spec requires
AudioTrack/VideoTrack.language to be valid BCP 47, but this isn't set in stone
if we don't think it makes sense.

In the case of TextTrack.language, the source of language can be <track
srclang="en">, but in this case there's no validation going on, see this demo:
http://software.hixie.ch/utilities/js/live-dom-viewer/saved/3514

[philipj_slow, 2015-04-29 14:02:20]

On 2015/04/29 13:59:29, Peter Beverloo wrote:
> There may well be use-cases, the i18n JavaScript API exposes and validates
> similar functionality in the "locales" arguments.

Interesting, I suppose that bit currently is not implemented in Blink, but is
there a spec I could look at? If that spec exposes this (for good reasons) then
it could make more sense to also use it in other places.

[Peter Beverloo, 2015-04-29 14:06:56]

On 2015/04/29 14:02:20, philipj_UTC2 wrote:
> On 2015/04/29 13:59:29, Peter Beverloo wrote:
> > There may well be use-cases, the i18n JavaScript API exposes and validates
> > similar functionality in the "locales" arguments.
> 
> Interesting, I suppose that bit currently is not implemented in Blink, but is
> there a spec I could look at? If that spec exposes this (for good reasons)
then
> it could make more sense to also use it in other places.

The implementation lives in v8: v8/src/i18n.js (with a scary regexp in
BuildLanguageTagREs()).

The drafts can be found here:
  http://wiki.ecmascript.org/doku.php?id=globalization:specification_drafts

It writes quite a bit about BCP 47 language tags, mostly in regards to
validation.

[philipj_slow, 2015-05-06 12:04:56]

Ping wolenetz@, any thoughts about the MSE situation?

[wolenetz, 2015-05-12 23:11:57]

On 2015/05/06 12:04:56, philipj_UTC2 wrote:
> Ping wolenetz@, any thoughts about the MSE situation?

Hmm. Neither could I find any strong discussion against having BCP 47 validation
in MSE. I think it chained from HTML spec that was extended. The interesting
portion of MSE is how to map an inband language to a BCP-47 language tag, and
how to best keep that aligned with src= inband attribute sourcing. Of the 4 byte
streams currently in the MSE registry, 3 specify how to source these by
providing a link to http://dev.w3.org/html5/html-sourcing-inband-tracks/. Within
that doc, I found no direct link to BCP-47 (though I haven't dug into linked
specs). Hence, I'm unsure if the sourcing instructions will always result in
either a "BCP 47 language tag" or an empty string, but not an invalid BCP 47
tag. And the MSE spec doesn't describe what to do in that case! For example, the
following language in the spec does not allow for an invalid BCP 47 language
tag, so either the sourcing must produce one, or there is some magic table that
maps arbitrary real-world stuff to a BCP 47 language tag:
"Let audio language be a BCP 47 language tag for the language specified in the
initialization segment for this track or an empty string if no language info is
present."

Given the uncertainty, I'm fine with this CL enforcing the current TrackDefault
constructor restriction to valid BCP 47 language tag (or empty string), but I'm
unsure (not in this CL) what to implement around patching up potentially
badly-sourced invalid BCP 47.

[philipj_slow, 2015-05-21 15:54:07]

On 2015/05/12 23:11:57, wolenetz wrote:
> On 2015/05/06 12:04:56, philipj_UTC2 wrote:
> > Ping wolenetz@, any thoughts about the MSE situation?
> 
> Hmm. Neither could I find any strong discussion against having BCP 47
validation
> in MSE. I think it chained from HTML spec that was extended. The interesting
> portion of MSE is how to map an inband language to a BCP-47 language tag, and
> how to best keep that aligned with src= inband attribute sourcing. Of the 4
byte
> streams currently in the MSE registry, 3 specify how to source these by
> providing a link to http://dev.w3.org/html5/html-sourcing-inband-tracks/.
Within
> that doc, I found no direct link to BCP-47 (though I haven't dug into linked
> specs). Hence, I'm unsure if the sourcing instructions will always result in
> either a "BCP 47 language tag" or an empty string, but not an invalid BCP 47
> tag. And the MSE spec doesn't describe what to do in that case! For example,
the
> following language in the spec does not allow for an invalid BCP 47 language
> tag, so either the sourcing must produce one, or there is some magic table
that
> maps arbitrary real-world stuff to a BCP 47 language tag:
> "Let audio language be a BCP 47 language tag for the language specified in the
> initialization segment for this track or an empty string if no language info
is
> present."
> 
> Given the uncertainty, I'm fine with this CL enforcing the current
TrackDefault
> constructor restriction to valid BCP 47 language tag (or empty string), but
I'm
> unsure (not in this CL) what to implement around patching up potentially
> badly-sourced invalid BCP 47.

I assume that most file formats express languages as a free-form string, so it
would be nice if the treatment were the same. That is, the strings from the
in-band tracks are passed verbatim to Blink and the strings set on TrackDefault
are forwarded verbatim. Both would go through the same layer of normalization
(if any) before being exposed on AudioTrack and VideoTrack objects. Although I'd
probably be happy with no normalization at all, just exposing that invalid to
scripts and saying "unknown language" in any UI based on it would work fine.

I'm not sure where this leaves us. I'm not the MSE spec editor, wolenetz@ is. I
am the mediasource module owner, but really wolenetz@ should be :) I'll defer to
your judgement. If none of us feel very strongly maybe we could poke blink-dev
about the wisdom of exposing BCP 47 validity to the Web.

BTW, I poked a bit at the i18n API and it does seems to follow the syntax of
http://tools.ietf.org/html/bcp47#section-2.1, but since only checks the syntax
and not any registry any word between 2-8 characters will pass, so new
Intl.DateTimeFormat(["English"]).format(Date.now()) passes the test even though
we couldn't extract the language from it.

[Peter Beverloo, 2015-07-15 10:08:20]

To update this -

While the i18n group definitely was in favor of confirming that language tags
were "well-formed" (as opposed to "valid", which is much harder to reliably
confirm), we decided to make this an author-level concern in the Notification
spec here:

https://github.com/whatwg/notifications/commit/743cd90906c5fa522d31ef0082b2a6...

Other specs might want to do something similar, also because we don't feel
strongly.

I'm abandoning this series of CLs.