The first step towards supporting the Hungarian spell-checking dictionary....

The first step towards supporting the Hungarian spell-checking dictionary.
This change fixes a couple of problems needed for using a Hungarian dictionary in Chrome.
1. Use TrimWhitespace() in TrimLine()
Sorry, this is caused by my mistake that used TrimWhiteSpaceUTF8() without checking it deeply.
2. Replace morphing rules with compound rules.
it seems existing Hungarian dictionaries use (language-specific) morphing rules to handle words that have both prefixes and suffixes, e.g. "legjobb" (best). It is better to replace such (language-dependent) morphing rules with (language-independent) compound rules to avoid language-specific issues. (As far as I tested, this change fixes many quality problems caused by Hungarian compounds.)
This change also adds simple tests for our dictionary converter.

BUG=15558
TEST=unit_test --gtest_filter=ConvertDictTest*

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=37816

[Hironori Bono, 2010-02-01 10:57:59]

ping?

[brettw, 2010-02-01 17:42:03]

LGTM.

http://codereview.chromium.org/553087/diff/6002/7002
File chrome/tools/convert_dict/aff_reader.cc (right):

http://codereview.chromium.org/553087/diff/6002/7002#newcode213
chrome/tools/convert_dict/aff_reader.cc:213: #if 0
Can you provide a comment about why this is ifdefed out? I assume it's because
Hungarian support isn't complete yet, so you should say this.

http://codereview.chromium.org/553087/diff/6002/7003
File chrome/tools/convert_dict/convert_dict_unittest.cc (right):

http://codereview.chromium.org/553087/diff/6002/7003#newcode176
chrome/tools/convert_dict/convert_dict_unittest.cc:176: // Tests whether or not
our DicReader can read all the input HUngarian words.
Uncapitalize 'U'

[Hironori Bono, 2010-02-02 10:04:15]

Brett,

Thank you for your review and comments.
I have applied your comments and committed the updated one as r37816.

Regards,

Hironori Bono

http://codereview.chromium.org/553087/diff/6002/7002
File chrome/tools/convert_dict/aff_reader.cc (right):

http://codereview.chromium.org/553087/diff/6002/7002#newcode213
chrome/tools/convert_dict/aff_reader.cc:213: #if 0
On 2010/02/01 17:42:03, brettw wrote:
> Can you provide a comment about why this is ifdefed out? I assume it's because
> Hungarian support isn't complete yet, so you should say this.

Thank you for noticing this.
This is just a bonehead mistake of mine. I forgot removing this #ifdef and
#endif lines when I created dictionaries with or without this code to compare
their quality. I have removed these lines since this code improves the quality.

http://codereview.chromium.org/553087/diff/6002/7003
File chrome/tools/convert_dict/convert_dict_unittest.cc (right):

http://codereview.chromium.org/553087/diff/6002/7003#newcode176
chrome/tools/convert_dict/convert_dict_unittest.cc:176: // Tests whether or not
our DicReader can read all the input HUngarian words.
On 2010/02/01 17:42:03, brettw wrote:
> Uncapitalize 'U'

Done.