Fix word count issue for Chinese and Japanese

java/org/chromium/distiller/StringUtil.java

Index: java/org/chromium/distiller/StringUtil.java
diff --git a/java/org/chromium/distiller/StringUtil.java b/java/org/chromium/distiller/StringUtil.java
index 00f387644ac3e50ab9ff091d8e21ee4dbc22a23c..9c0d081434f04a5d239bc651471f0710557bd55c 100644
--- a/java/org/chromium/distiller/StringUtil.java
+++ b/java/org/chromium/distiller/StringUtil.java
@@ -47,8 +47,14 @@ public class StringUtil {
     }-*/;
 
     public static native int countWords(String s) /*-{

[cjhopman, 2015/05/15 20:16:55]

Does this new approach make sense everywhere that we use countWords?

[cjhopman, 2015/05/15 20:16:55]

Maybe we should change the name of this to reflect that it's not really counting
words anymore.

[wychen, 2015/05/18 18:49:20]

Well, the goal of this function is still to count words, but it is not feasible
to accurately count words for some languages. For those difficult languages,
character count is used as an approximation (adjusted by a factor). I've added
comments to this function to explain this.

[wychen, 2015/05/18 18:49:20]

One problem I see is the title finding part. It contains some magic numbers that
are inferred from English corpus, which should still work for Chinese and
Japanese after this CL. We might want to make it more robust in a separate CL
though, and also enhance eval server and dataset to validate that change. Or
validate that our current magic numbers are actually robust enough.

-        var m = s.match(/(\S*[\w\u00C0-\u1FFF\u2C00-\uD7FF]\S*)/g);
-        return m ? m.length : 0;
+        // The following range includes broader alphabetical letters and Hangul Syllables.
+        var m = s.match(/(\S*[\w\u00C0-\u1FFF\uAC00-\uD7AF]\S*)/g);
+        var c = (m ? m.length : 0);
+        // The following range includes Hiragana, Katakana, and CJK Unified Ideographs.
+        // Hangul Syllables are not included.
+        m = s.match(/([\u3040-\uA4CF])/g);
+        c += Math.ceil((m ? m.length : 0) * 0.55);
+        return c;
     }-*/;
 
     public static native String regexEscape(String s) /*-{