Fix word count issue for Chinese and Japanese

java/org/chromium/distiller/StringUtil.java

 // Copyright 2014 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 package org.chromium.distiller;
 
 import com.google.gwt.regexp.shared.RegExp;
 
 public class StringUtil {
     // For the whitespace-related functions below, Java's and Javascript's versions of '\s' and '\S'
@@ skipping 24 matching lines @@
     }
 
     public static boolean match(String input, String regex) {
         return RegExp.compile(regex, "i").test(input);
     }
 
     public static String findAndReplace(String input, String regex, String replace) {
         return RegExp.compile(regex, "gi").replace(input, replace);
     }
 
-    public static native boolean containsWordCharacter(String s) /*-{
-        return /[\w\u00C0-\u1FFF\u2C00-\uD7FF]/.test(s);
+    /**
+     * For some languages, counting the number of words relies on non-trivial word
+     * segmentation algorithms, or even huge look-up tables. This function needs to
+     * be reasonably fast, so the word count for some languages would only be an
+     * approximation.
+     * Read https://crbug.com/484750 for more info.
+     */
+
+    // Placeholder for the local function variable. The type is wrong.
+    public static int _countWords;

[cjhopman, 2015/05/26 21:53:06]

Can't this be a JavaScriptObject? Those can store javascript functions just
fine.

Alternatively, we could do this the real Java way:

Have an interface for this, and have three implementations of it that each have
a different jsni implementation of the one function. Then just initialize a
member of that interface.

[wychen, 2015/05/29 07:44:09]

Done the real Java way.

+
+    public static native void selectCountWordsFunc(String text) /*-{
+        if (/[\u3040-\uA4CF]/.test(text)) {
+            @StringUtil::_countWords = function(s) {
+                // The following range includes broader alphabetical letters and Hangul Syllables.
+                var m = s.match(/(\S*[\w\u00C0-\u1FFF\uAC00-\uD7AF]\S*)/g);
+                var c = (m ? m.length : 0);
+                // The following range includes Hiragana, Katakana, and CJK Unified Ideographs.
+                // Hangul Syllables are not included.
+                m = s.match(/([\u3040-\uA4CF])/g);
+                c += Math.ceil((m ? m.length : 0) * 0.55);
+                return c;
+            };
+            return;
+        }
+        if (/[\AC00-\uD7AF]/.test(text)) {
+            @StringUtil::_countWords = function(s) {
+                // The following range includes broader alphabetical letters and Hangul Syllables.
+                var m = s.match(/(\S*[\w\u00C0-\u1FFF\uAC00-\uD7AF]\S*)/g);
+                return (m ? m.length : 0);
+            };
+            return;
+        }
+        @StringUtil::_countWords = function(s) {
+            // The following range includes broader alphabetical letters.
+            var m = s.match(/(\S*[\w\u00C0-\u1FFF]\S*)/g);
+            return (m ? m.length : 0);
+        };
     }-*/;
 
+    // Use the safest version of countWords as the default.
+    private static final int __dummy_initCountWords = initCountWords();
+    private static int initCountWords() {
+        selectCountWordsFunc("字");
+        return 0;
+    }
+
     public static native int countWords(String s) /*-{
-        var m = s.match(/(\S*[\w\u00C0-\u1FFF\u2C00-\uD7FF]\S*)/g);
-        return m ? m.length : 0;
+        // Use a different variable to get around the type checking.
+        var func = @StringUtil::_countWords;
+        return func(s);
     }-*/;
 
     public static native String regexEscape(String s) /*-{
         return s.replace(/[\-\[\]{}()*+?.,\\\^$|#\s]/g, "\\$&");
     }-*/;
 
     /*
      * Returns true if character is a digit.
      */
     public static native boolean isDigit(Character c) /*-{
@@ skipping 17 matching lines @@
     /**
      * Returns the plain number if given string can be converted to one >= 0.
      * Returns -1 if string is empty or not all digits.
      */
     public static int toNumber(String s) {
         if (s.isEmpty() || !StringUtil.isStringAllDigits(s)) return -1;
         return JavaScript.parseInt(s, 10);
     }
 
 }