Port back CLD to Linux and Mac and fix Trad Chinese detection

Port back CLD to Linux and Mac by replacing Windows API calls with ICU's equivalent APIs for normalization and lowercasing.

I also fixed bug 23553 (Traditional Chinese is not detected) by calling LanguageCode instead of LanguageCode_ISO_639_1 (the latter covers only ISO-639-1, but there are languages detected by CLD that are not covered by ISO 639-1 codes). 

The html file for CLD testing (french_sentence.html) is explicitly labelled with
charset=ISO-8859-1.

BUG=25206, 23553
TEST=1. CLD is built on Linux/Mac and the following test pass:
   - unit_tests:  Extension*.DetectTabLanguage and CompactLangDetTest.*
   - browser_tests: ExtensionBrowserTest.Toolstrip
2. Install the 'cld extension' in chrome/common/extensions/docs/examples/api/i18n/cld and go to http://news.google.com.tw. 'zh-TW' should show up in the upper right hand corner instead of '???'. 

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=36541

[jungshik at Google, 2010-01-12 23:37:53]

Jay, can you take a look? 


DetectLanguageOfUnicodeText can be changed to accept string16 intead of UChar*
(which is unsigned short*), in which case string16.h header has to be added to
third_party/cld/base.  

Further clean-up can be done (e.g. I don't think we need to carry an
unnecessarily long path like bar/toolbar/i18n) and it can be made stand-alone
ala googleurl. (alternatively, it can be folded to ICU, which was discussed a
long while ago with the original author of CLD). Anyway, this can work for now.

[jcampan, 2010-01-13 00:24:18]

LGTM with few small nits.

Thanks for porting this!
Note that I moved the CLD detection to the renderer, so you'll run into
conflicts next time you sync.

http://codereview.chromium.org/523108/diff/15010/14010
File
third_party/cld/bar/toolbar/cld/i18n/encodings/compact_lang_det/win/cld_unicodetext.cc
(right):

http://codereview.chromium.org/523108/diff/15010/14010#newcode11
third_party/cld/bar/toolbar/cld/i18n/encodings/compact_lang_det/win/cld_unicodetext.cc:11:
//#include "bar/toolbar/cld/i18n/encodings/compact_lang_det/win/cld_scopedptr.h"
Nit: Remove

http://codereview.chromium.org/523108/diff/15010/14010#newcode23
third_party/cld/bar/toolbar/cld/i18n/encodings/compact_lang_det/win/cld_unicodetext.cc:23:
return std::string();
Indent should be 2 spaces

http://codereview.chromium.org/523108/diff/15010/14010#newcode47
third_party/cld/bar/toolbar/cld/i18n/encodings/compact_lang_det/win/cld_unicodetext.cc:47:
// Normalize text to NFC, lowercase and convert to UTF-8
Nit: add period at end of comment.

[jungshik at Google, 2010-01-13 07:11:21]

Thanks for the review and the note about the language detection moving to the
renderer. 

I removed the change for the browser process and changed the renderer, instead
after updating my tree. Somehow, the browser test for the language detection
fails. I'll debug to see what's going on. 

In addition, I took care of nits.

[jungshik at Google, 2010-01-19 19:31:28]

Jay, can you take another look? There have been some changes that may warrant
another look. 

1. Fixed my stupid mistake with ICU error handling, which made the browser_test
fail initially in cld_unicodetext.

2. While I was at it, I switched to |toUTF8String| method of
|icu::UnicodeString|. It does more or less the same thing as my initial code
(using low-level ICU apis and mpcomplete's WriteInto) although how and when
malloc/new is called is different. 

3. Mac/Linux still failed to work with my change (even though
'not-included-in-this CL' unittest of DetectLanguageOfUnicodeText passed, which
baffled me. It turned out that I removed only one of #ifdef WIN in
render_view.cc.  As a result, on Mac/Linux, kUndefinedLanguageCode is always
returned :-). 

4. Fixing #3 led me to find the cause of bug 23553 (Traditional Chinese being
not detected), which was fixed by calling LanguageCode (a more generic than
LanguageCode_ISO_639_1 that is currently used.). 

5. When a language cannot be defined, 'und' (ISO 639 code for undefinded
language) is used instead of 'undefined'. 

For bug 23553, I can add a browser test with 'traditonal_chinse.html' (as is
done for 'french_sentence.html'). An alternative is to add a function 
|DetectLanguageCodeOfUnicodeText| to cld_unicodetext.cc and add a unit test for
that. To me, the latter seems better (it can share the test data for
CompactLangDetTest.*.). If you agree, I'll follow up this CL with that (and fix
for  http://crbug.com/20033).

[jcampan, 2010-01-19 19:42:58]

LGTM