Unpunycode search keywords and short names.

components/search_engines/template_url.cc

 // Copyright 2014 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 #include "components/search_engines/template_url.h"
 
 #include <string>
 #include <vector>
 
 #include "base/basictypes.h"
@@ skipping 1199 matching lines @@
 }
 
 // static
 base::string16 TemplateURL::GenerateKeyword(const GURL& url) {
   DCHECK(url.is_valid());
   // Strip "www." off the front of the keyword; otherwise the keyword won't work
   // properly.  See http://code.google.com/p/chromium/issues/detail?id=6984 .
   // Special case: if the host was exactly "www." (not sure this can happen but
   // perhaps with some weird intranet and custom DNS server?), ensure we at
   // least don't return the empty string.
-  base::string16 keyword(net::StripWWWFromHost(url));
+  // We do not want any punycode in any language.
+  base::string16 keyword(net::StripWWW(net::IDNToUnicode(url.host(), "")));

[Peter Kasting, 2015/07/14 18:05:33]

I think we should pass the user's accept-languages here, so we'll un-punycode
more cases that the user should understand.

This should probably be passed to this function as a second arg, and so on up
the call chain until you get to a chrome-side caller that can compute this as:

  profile_->GetPrefs()->GetString(prefs::kAcceptLanguages)

Another route in some cases would be to add an embedder client method to get
these languages, then have the components-side code that needs to call the
client method get the embedder to instantiate such a client instance for it. 
Yet another route would be to add this information, or accessors that compute
it, to SearchTermsData.

[alshabalin, 2015/07/15 14:18:01]

My thought was that a string in a language user does not read and a string in
IDN form are equally incomprehensible but the first one looks nicer. After all,
punycode is there only because hostname must be ASCII and as such is an
implementation detail.

Or is there some guideline that we should unpunycode only to the languages
the user understands?

[Peter Kasting, 2015/07/15 21:19:12]

It seems like you're misunderstanding how the conversion works.  If you pass no
languages, we will fail to render as Unicode any hostnames that mix characters
from different scripts.  If you pass the languages the user understands, then
we'll be willing to show as Unicode hostnames which use characters from those
scripts.  Passing the empty string doesn't mean "un-punycode everything", it
means something closer to the opposite.

In other words, I'm trying to get your code to un-punycode more hostnames, not
fewer.

[alshabalin, 2015/07/16 15:13:07]

I thought script-mixing in hostnames is not allowed anyway because it opens up
more ways to spoof. But apparently I'm wrong
https://tools.ietf.org/html/rfc5890#section-4.4
My mistake, sorry, will rewrite to pass accepted languages then.

[alshabalin, 2015/07/20 14:06:37]

Done.

   return keyword.empty() ? base::ASCIIToUTF16("www") : keyword;
 }
 
 // static
 GURL TemplateURL::GenerateFaviconURL(const GURL& url) {
   DCHECK(url.is_valid());
   GURL::Replacements rep;
 
   const char favicon_path[] = "/favicon.ico";
   int favicon_path_len = arraysize(favicon_path) - 1;
@@ skipping 278 matching lines @@
       // patterns. This means that given patterns
       //    [ "http://foo/#q={searchTerms}", "http://foo/?q={searchTerms}" ],
       // calling ExtractSearchTermsFromURL() on "http://foo/?q=bar#q=' would
       // return false. This is important for at least Google, where such URLs
       // are invalid.
       return !search_terms->empty();
     }
   }
   return false;
 }