Adds ChineseScriptClassifier to predict zh-Hant or zh-Hans for input detected as zh.

components/translate/core/language_detection/chinese_script_classifier.cc

+// Copyright 2017 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include "components/translate/core/language_detection/chinese_script_classifier.h"
+
+#include <algorithm>
+#include <cctype>
+#include <memory>
+#include <string>
+#include "base/logging.h"
+#include "base/strings/string_util.h"
+#include "third_party/icu/source/common/unicode/unistr.h"
+#include "third_party/icu/source/i18n/unicode/translit.h"
+
+namespace translate {
+
+const char ChineseScriptClassifier::kChineseSimplifiedCode[] = "zh-Hans";
+const char ChineseScriptClassifier::kChineseTraditionalCode[] = "zh-Hant";
+
+ChineseScriptClassifier::ChineseScriptClassifier() {
+  UParseError parse_status;
+  UErrorCode status = U_ZERO_ERROR;
+  // The Transliterator IDs are defined in:
+  // third_party/icu/source/data/translit/root.txt.
+  //
+  // Chromium keeps only a subset of these, defined in:
+  // third_party/icu/source/data/translit/root_subset.txt
+  hans2hant_.reset(icu::Transliterator::createInstance(
+      icu::UnicodeString("Hans-Hant"), UTRANS_FORWARD, parse_status, status));
+  VLOG(1) << "Hans-Hant Transliterator initialization status: "
+          << u_errorName(status);
+  hant2hans_.reset(icu::Transliterator::createInstance(
+      icu::UnicodeString("Hant-Hans"), UTRANS_FORWARD, parse_status, status));
+  VLOG(1) << "Hant-Hans Transliterator initialization status: "
+          << u_errorName(status);
+}
+
+bool ChineseScriptClassifier::IsInitialized() const {
+  return hans2hant_ && hant2hans_;
+}
+
+ChineseScriptClassifier::~ChineseScriptClassifier() {}
+
+std::string ChineseScriptClassifier::Classify(const std::string& input) const {
+  // If there was a problem with initialization, return the empty string.
+  if (!IsInitialized()) {
+    return "";
+  }
+
+  // Operate only on first 500 bytes.
+  std::string input_subset;
+  base::TruncateUTF8ToByteSize(input, 500, &input_subset);
+
+  // Remove whitespace since transliterators may not preserve it.
+  input_subset.erase(std::remove_if(input_subset.begin(), input_subset.end(),
+                                    base::IsUnicodeWhitespace),
+                     input_subset.end());
+
+  // Convert two copies of the input to icu::UnicodeString. Two copies are
+  // necessary because transliteration happens in place only.
+  icu::UnicodeString original_input =
+      icu::UnicodeString::fromUTF8(input_subset);
+  icu::UnicodeString hant_input = icu::UnicodeString::fromUTF8(input_subset);
+  icu::UnicodeString hans_input = icu::UnicodeString::fromUTF8(input_subset);
+
+  // Get the zh-Hant version of this input.
+  hans2hant_->transliterate(hant_input);
+  // Get the zh-Hans version of this input.
+  hant2hans_->transliterate(hans_input);
+
+  // Debugging only: show the input, the Hant version, and the Hans version.
+  if (VLOG_IS_ON(1)) {
+    std::string hant_string;
+    std::string hans_string;
+    hans_input.toUTF8String(hans_string);
+    hant_input.toUTF8String(hant_string);
+    VLOG(1) << "Original input:\n" << input_subset;

[groby-ooo-7-16, 2017/03/08 00:41:54]

Are you planning to keep the VLOG forever, or is there an expiration date?

[riesa, 2017/03/08 01:47:42]

Yes, I was planning to keep it in case of bug reports. But it sounds like
keeping these are discouraged in Chromium?

[groby1, 2017/03/08 02:00:57]

I'm torn. Each VLOG increases binary size, which affects how easy it is for
people to download Chrome. (Ping me if you'd like details)

But I do see that they'd be helpful for debugging, although it requires
modifying the cmdline to even see output. Would DVLOG be OK?

+    VLOG(1) << "zh-Hant output:\n" << hant_string;
+    VLOG(1) << "zh-Hans output:\n" << hans_string;
+  }
+
+  // Count matches between the original input chars and the Hant and Hans
+  // versions of the input.
+  int hant_count = 0;
+  int hans_count = 0;
+  for (int index = 0;
+       index < original_input.length() && index < hans_input.length() &&

[groby-ooo-7-16, 2017/03/08 00:41:54]

nit: please do compute min outside loop

[riesa, 2017/03/08 01:47:42]

Done.

+       index < hant_input.length();
+       ++index) {
+    const auto original_char = original_input.charAt(index);
+    const auto hans_char = hans_input.charAt(index);

[groby-ooo-7-16, 2017/03/08 00:41:54]

Bit concerned by the fact that charAt needs to repeatedly traverse the entire
string up to index. Do we have UTF8 iterators?

[riesa, 2017/03/08 01:47:42]

Hm, from what I can tell charAt is just indexing right into a UChar (a 16-bit
wide char) array.
https://cs.chromium.org/chromium/src/third_party/icu/source/common/unicode/un...

[groby1, 2017/03/08 02:00:57]

Gah. I misread the types to be still UTF8. Never mind then :) (Modern CJK chars
all live in the BMP, correct?)

+    const auto hant_char = hant_input.charAt(index);
+    if (hans_char == hant_char) {

[groby-ooo-7-16, 2017/03/08 00:41:54]

Possible simplification (no braces, less branches, no continue)

if (original_char == hans_char)
  ++hans_count;
if (original_char == hant_char)
  ++hant_count;

[riesa, 2017/03/08 01:47:42]

I originally had it this way in an earlier patch, and I had been going back and
forth on whether to go back to it.

The current if-structure makes debugging easier when we know exactly how many
and which script-unique chars to look for; these are what determine the
classification. If we have made it past the (hans_char == hant_char) check, then
every time I increment I know I found something unique. In the simpler case
which is functionally and productionally equivalent, I don't know this
information.

Since this code will be run many more times in production than I will run it to
debug something, and since you had a similar thought, then I will change it
back.

[groby1, 2017/03/08 02:00:57]

I don't think there's a performance penalty worth considering. If this is
cleaner to you and helps debugging, keep it. I just have a bias towards shorter
code :)

+      continue;
+    } else if (original_char == hans_char) {
+      // Input matches a Hans-only char.
+      ++hans_count;
+    } else if (original_char == hant_char) {
+      // Input matches a Hant-only char.
+      ++hant_count;
+    }
+  }
+  VLOG(1) << "Found " << hans_count << " zh-Hans chars in input";
+  VLOG(1) << "Found " << hant_count << " zh-Hant chars in input";
+
+  if (hant_count > hans_count) {
+    return kChineseTraditionalCode;
+  } else if (hans_count > hant_count) {
+    return kChineseSimplifiedCode;
+  } else {  // hans_count == hant_count
+    // All characters are the same in both scripts. In this case, we return the
+    // following code.
+    return kChineseSimplifiedCode;
+  }
+}
+
+}  // namespace translate