Fix word count issue for Chinese and Japanese

java/org/chromium/distiller/StringUtil.java

 // Copyright 2014 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 package org.chromium.distiller;
 
 import com.google.gwt.regexp.shared.RegExp;
 
 public class StringUtil {
     // For the whitespace-related functions below, Java's and Javascript's versions of '\s' and '\S'
@@ skipping 24 matching lines @@
     }
 
     public static boolean match(String input, String regex) {
         return RegExp.compile(regex, "i").test(input);
     }
 
     public static String findAndReplace(String input, String regex, String replace) {
         return RegExp.compile(regex, "gi").replace(input, replace);
     }
 
     public static native boolean containsWordCharacter(String s) /*-{

[cjhopman, 2015/05/15 20:16:55]

We need to ensure that

containsWordCharacter(s) == (countWords(s) > 0)

for all strings. 

If containsWordCharacter isn't used, we could just delete it.

[wychen, 2015/05/18 18:49:20]

Deleted.

         return /[\w\u00C0-\u1FFF\u2C00-\uD7FF]/.test(s);
     }-*/;
 
     public static native int countWords(String s) /*-{

[cjhopman, 2015/05/15 20:16:55]

Does this new approach make sense everywhere that we use countWords?

[cjhopman, 2015/05/15 20:16:55]

Maybe we should change the name of this to reflect that it's not really counting
words anymore.

[wychen, 2015/05/18 18:49:20]

Well, the goal of this function is still to count words, but it is not feasible
to accurately count words for some languages. For those difficult languages,
character count is used as an approximation (adjusted by a factor). I've added
comments to this function to explain this.

[wychen, 2015/05/18 18:49:20]

One problem I see is the title finding part. It contains some magic numbers that
are inferred from English corpus, which should still work for Chinese and
Japanese after this CL. We might want to make it more robust in a separate CL
though, and also enhance eval server and dataset to validate that change. Or
validate that our current magic numbers are actually robust enough.

-        var m = s.match(/(\S*[\w\u00C0-\u1FFF\u2C00-\uD7FF]\S*)/g);
-        return m ? m.length : 0;
+        // The following range includes broader alphabetical letters and Hangul Syllables.
+        var m = s.match(/(\S*[\w\u00C0-\u1FFF\uAC00-\uD7AF]\S*)/g);
+        var c = (m ? m.length : 0);
+        // The following range includes Hiragana, Katakana, and CJK Unified Ideographs.
+        // Hangul Syllables are not included.
+        m = s.match(/([\u3040-\uA4CF])/g);
+        c += Math.ceil((m ? m.length : 0) * 0.55);
+        return c;
     }-*/;
 
     public static native String regexEscape(String s) /*-{
         return s.replace(/[\-\[\]{}()*+?.,\\\^$|#\s]/g, "\\$&");
     }-*/;
 
     /*
      * Returns true if character is a digit.
      */
     public static native boolean isDigit(Character c) /*-{
@@ skipping 17 matching lines @@
     /**
      * Returns the plain number if given string can be converted to one >= 0.
      * Returns -1 if string is empty or not all digits.
      */
     public static int toNumber(String s) {
         if (s.isEmpty() || !StringUtil.isStringAllDigits(s)) return -1;
         return JavaScript.parseInt(s, 10);
     }
 
 }