Extend with a language code in http accept languages

net/http/http_util_icu.cc

Index: net/http/http_util_icu.cc
diff --git a/net/http/http_util_icu.cc b/net/http/http_util_icu.cc
new file mode 100644
index 0000000000000000000000000000000000000000..6c5d33117d7f6ac386fecb49dc5dff4255219b03
--- /dev/null
+++ b/net/http/http_util_icu.cc
@@ -0,0 +1,97 @@
+// Copyright (c) 2016 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include "net/http/http_util_icu.h"
+
+#include <algorithm>
+#include <set>
+#include <string>

[Seigo Nonaka, 2016/12/12 07:29:26]

string is necessary for std::string

[Yirui Huang, 2016/12/12 07:41:35]

Done.

+#include <vector>
+
+#include "base/logging.h"
+#include "base/strings/string_number_conversions.h"
+#include "base/strings/string_piece.h"
+#include "base/strings/string_split.h"
+#include "base/strings/string_tokenizer.h"
+#include "base/strings/string_util.h"
+#include "base/strings/stringprintf.h"
+#include "base/time/time.h"

[Seigo Nonaka, 2016/12/12 01:48:54]

probably, some of headers are not necessary, e.g. time.h.
Please minimize the header inclusions.

[Yirui Huang, 2016/12/12 06:50:41]

Done.

+#include "net/base/url_util.h"
+#include "third_party/icu/source/common/unicode/uloc.h"
+
+namespace net {
+
+// The input is a comma separated languages list, this function allows
+// whitespace between each languages. As long as it comes from the preference
+// and a user does not manually edit the preference file, it's the case.
+// TODO(yirui): We still need to be aware of cases, make it more robust.

[Seigo Nonaka, 2016/12/12 01:48:54]

AndroidWebView is the only case that the developer override the accept language.
Could you kindly check what string is passed here (or simply this is not called)
with following case.

Map<String, String> headers = new HashMap<>();
headers.put("Accept-Language", "xxx,yyy;q=0.8");
webview.loadUrl("http://www.google.com", headers);

If the passed accept language string is passed without any modification, we may
need to fix this function. However, even in that case, the existing code also
has the same issue, so please feel free to leave TODO.

[Yirui Huang, 2016/12/12 06:50:40]

The change from developers will not use http_util, therefore, raw string will be
in well form.

+std::string HttpUtilIcu::GenerateAcceptLanguageHeader(
+    const std::string& raw_language_list) {
+  std::vector<std::string> locales_list = base::SplitString(
+      raw_language_list, ",", base::TRIM_WHITESPACE, base::SPLIT_WANT_NONEMPTY);
+
+  // If language is not in the accept languages list, also add language
+  // code. A language code should only be inserted after the last
+  // languageTag that contains that language.
+  // This will work with the IDS_ACCEPT_LANGUAGE localized strings bundled

[Seigo Nonaka, 2016/12/12 01:48:54]

I think this comment is no longer necessary.

[Yirui Huang, 2016/12/12 06:50:41]

Done.

+  // with Chrome but may fail on arbitrary lists of language tags due to
+  // differences in case and whitespace.
+  std::set<std::string> seen_languages;
+  std::vector<std::string> output_list;
+  for (auto it = locales_list.rbegin(); it != locales_list.rend(); ++it) {
+    char locale_ID[ULOC_FULLNAME_CAPACITY] = {};
+    char language_code[ULOC_LANG_CAPACITY] = {};
+
+    UErrorCode error = U_ZERO_ERROR;
+    std::string locale_string = *it;
+    uloc_forLanguageTag(locale_string.c_str(), locale_ID,
+                        ULOC_FULLNAME_CAPACITY, nullptr, &error);
+    if (U_FAILURE(error)) {
+      LOG(ERROR) << "Ignoring invalid locale representation " << locale_string;
+      continue;
+    }
+
+    error = U_ZERO_ERROR;
+    uloc_getLanguage(locale_ID, language_code, ULOC_LANG_CAPACITY, &error);
+    if (U_FAILURE(error)) {
+      LOG(ERROR) << "Ignoring invalid locale representation " << locale_string;
+      continue;
+    }
+
+    if (seen_languages.find(language_code) == seen_languages.end()) {
+      output_list.push_back(language_code);
+      seen_languages.insert(language_code);
+    }
+
+    if (language_code != *it)
+      output_list.push_back(locale_string);
+  }
+
+  std::reverse(output_list.begin(), output_list.end());
+  std::string final_language_list = base::JoinString(output_list, ",");
+
+  // We use integers for qvalue and qvalue decrement that are 10 times
+  // larger than actual values to avoid a problem with comparing
+  // two floating point numbers.
+  const unsigned int kQvalueDecrement10 = 2;
+  unsigned int qvalue10 = 10;
+  base::StringTokenizer t(final_language_list, ",");
+  std::string lang_list_with_q;
+  while (t.GetNext()) {

[Seigo Nonaka, 2016/12/12 01:48:54]

Let's use output_list here. You joined string then split it again.

[Yirui Huang, 2016/12/12 06:50:40]

Done.

+    std::string language = t.token();
+    if (qvalue10 == 10) {
+      // q=1.0 is implicit.
+      lang_list_with_q = language;
+    } else {
+      DCHECK_LT(qvalue10, 10U);
+      base::StringAppendF(&lang_list_with_q, ",%s;q=0.%d", language.c_str(),
+                          qvalue10);
+    }
+    // It does not make sense to have 'q=0'.
+    if (qvalue10 > kQvalueDecrement10)
+      qvalue10 -= kQvalueDecrement10;
+  }
+  return lang_list_with_q;
+}
+}