Roll Compact Language Detector v2 to r156, the January 2014 release (+stuff!)

Roll Compact Language Detector v2 to r156, the January 2014 release (+stuff!)

This change will allow us to pick up much goodness:
1. Improved detection for most languages
2. New options for building CLD2 libraries of different sizes for each platform
3. Dynamic data mode (allows CLD's data to be extracted from the executable)
4. Support for 6 new languages
... and more. These are big changes. For more complete info:
https://code.google.com/p/cld2/wiki/January2014Release

This change also updates the .gyp file to pick up the new files and start using
them immediately. The old files should still work, but there's little point in
rolling if we're not going to take the new files.

BUG=

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=257102

[Andrew Hayden (chromium.org), 2014-03-12 10:50:55]

FWIW I have tested this with a local linux build and it works flawlessly. Not
shocking since there are no API changes here, but just noting for the record.

[Andrew Hayden (chromium.org), 2014-03-12 11:26:49]

Trybot on Windows isn't happy because the compiler there won't allow an array of
size zero:

FAILED: ninja -t msvc -e environment.x86 -- E:\b\build\goma\gomacc.exe
"E:\b\depot_tools\win_toolchain\vs2013_files\VC\bin\amd64_x86\cl.exe" /nologo
/showIncludes /FC
@obj\third_party\cld_2\src\internal\cld_2.cld2_generated_distinctoctachrome0122.obj.rsp
/c ..\..\third_party\cld_2\src\internal\cld2_generated_distinctoctachrome0122.cc
/Foobj\third_party\cld_2\src\internal\cld_2.cld2_generated_distinctoctachrome0122.obj
/Fdobj\third_party\cld_2\cld_2.cc.pdb 
e:\b\build\slave\win\build\src\third_party\cld_2\src\internal\cld2_generated_distinctoctachrome0122.cc(2184)
: error C2466: cannot allocate an array of constant size 0
e:\b\build\slave\win\build\src\third_party\cld_2\src\internal\cld2_generated_distinctoctachrome0122.cc(2186)
: error C2466: cannot allocate an array of constant size 0
FAILED: ninja -t msvc -e environment.x86 -- E:\b\build\goma\gomacc.exe
"E:\b\depot_tools\win_toolchain\vs2013_files\VC\bin\amd64_x86\cl.exe" /nologo
/showIncludes /FC
@obj\third_party\cld_2\src\internal\cld_2.cld2_generated_deltaoctachrome0122.obj.rsp
/c ..\..\third_party\cld_2\src\internal\cld2_generated_deltaoctachrome0122.cc
/Foobj\third_party\cld_2\src\internal\cld_2.cld2_generated_deltaoctachrome0122.obj
/Fdobj\third_party\cld_2\cld_2.cc.pdb 
e:\b\build\slave\win\build\src\third_party\cld_2\src\internal\cld2_generated_deltaoctachrome0122.cc(4577)
: error C2466: cannot allocate an array of constant size 0
e:\b\build\slave\win\build\src\third_party\cld_2\src\internal\cld2_generated_deltaoctachrome0122.cc(4579)
: error C2466: cannot allocate an array of constant size 0

This is something Dick had anticipated, but it worked under the linux gcc. We'll
have to find a workaround to make the compiler happy. Most likely we'll do what
we used to do, which is allocate an array of size 1 entry (4 bytes), but leave
the size constant set to zero such that at runtime the 4 bytes in the array are
completely unreferenced. Icky, but low impact.

[Andrew Hayden (chromium.org), 2014-03-12 11:36:32]

Fave filed a CLD2 issue to fix the windows compilation problem:
https://code.google.com/p/cld2/issues/detail?id=9

[Andrew Hayden (chromium.org), 2014-03-12 12:00:33]

Adjusted the code to eliminate the problematic zero-length arrays, returning to
an approach we used to use here that should make the compiler on windows happy.
Verified that all is still well, see the issue for more complete details. Trying
bots on r155 now.

[hajimehoshi, 2014-03-13 02:30:10]

Thank you! lgtm

[Andrew Hayden (chromium.org), 2014-03-13 17:16:26]

I realized that I had selected the small datasets:
  src/internal/cld2_generated_quadchrome0122_16.cc
  src/internal/cld_generated_cjk_delta_bi_4.cc

... but this isn't the right choice. Right now only the desktop builds use CLD2;
Android and iOS are still on CLD1. The smaller and slightly less accurate files
are intended specifically for mobile Chromium builds. So the right thing to do
is make the default to use the large data sets for now, and then in a subsequent
change we can parameterize the choice of size with a build variable and let
Android/iOS take the smaller files:
  src/internal/cld2_generated_quadchrome0122_16.cc
  src/internal/cld_generated_cjk_delta_bi_32.cc

Any objections? I'm ready to commit this once the bots are happy.

[dsites_google.com, 2014-03-13 17:23:59]

No need to use src/internal/cld_generated_cjk_delta_bi_32.cc, it is only
for the full-size 175-language version, not Chrome.
Just shift from
   src/internal/cld2_generated_quadchrome0122_16.cc
to
   src/internal/cld2_generated_quadchrome0122_2.cc

/dick



On Thu, Mar 13, 2014 at 10:16 AM, <andrewhayden@chromium.org> wrote:

> I realized that I had selected the small datasets:
>   src/internal/cld2_generated_quadchrome0122_16.cc
>   src/internal/cld_generated_cjk_delta_bi_4.cc
>
> ... but this isn't the right choice. Right now only the desktop builds use
> CLD2;
> Android and iOS are still on CLD1. The smaller and slightly less accurate
> files
> are intended specifically for mobile Chromium builds. So the right thing
> to do
> is make the default to use the large data sets for now, and then in a
> subsequent
> change we can parameterize the choice of size with a build variable and let
> Android/iOS take the smaller files:
>   src/internal/cld2_generated_quadchrome0122_16.cc
>   src/internal/cld_generated_cjk_delta_bi_32.cc
>
> Any objections? I'm ready to commit this once the bots are happy.
>
> https://codereview.chromium.org/197123002/
>

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[Andrew Hayden (chromium.org), 2014-03-13 17:26:46]

On 2014/03/13 17:23:59, dsites_google.com wrote:
> No need to use src/internal/cld_generated_cjk_delta_bi_32.cc, it is only
> for the full-size 175-language version, not Chrome.
> Just shift from
>    src/internal/cld2_generated_quadchrome0122_16.cc
> to
>    src/internal/cld2_generated_quadchrome0122_2.cc
> 
> /dick

Thanks, Dick. Done and patchset 4 uploaded.

[Andrew Hayden (chromium.org), 2014-03-14 09:00:21]

I had to make one more round of minor bugfixes for compilation on Windows. The
patch to CLD can be found here:
https://code.google.com/p/cld2/issues/detail?id=9

It was another instance of zero-length array allocation, and the fix is the same
as the move from r154 to r155. This has no visible effect other than to make the
compiler happy on Windows, and the data file remains a perfect binary copy since
r154.

[hajimehoshi, 2014-03-14 09:09:12]

still LGTM

[Andrew Hayden (chromium.org), 2014-03-14 09:10:12]

I am seeing some failures on the trybots in the SecurityTest.NewOverflow on mac.
This looks like it could be
https://code.google.com/p/chromium/issues/detail?id=227092 so I'm not too
worried yet.

[Andrew Hayden (chromium.org), 2014-03-14 12:49:19]

The CQ bit was checked by andrewhayden@chromium.org

[commit-bot: I haz the power, 2014-03-14 12:49:21]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/andrewhayden@chromium.org/197123002/11...

[commit-bot: I haz the power, 2014-03-14 12:50:40]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-03-14 12:50:41]

Try jobs failed on following builders:
  tryserver.chromium on linux_chromium_clang_dbg

[Andrew Hayden (chromium.org), 2014-03-14 12:53:47]

The CQ bit was checked by andrewhayden@chromium.org

[commit-bot: I haz the power, 2014-03-14 12:53:57]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/andrewhayden@chromium.org/197123002/11...

[commit-bot: I haz the power, 2014-03-14 14:41:46]

Change committed as 257102