Make it possible to read CLD data from a file

third_party/cld_2/cld_2.gyp

 # Copyright 2013 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 # Note to maintainers: In the January 2014 release (*_0122*), there are some
 # options for building:
 # Pick one quadgram file (cld2_generated_quadchrome*.cc):
 #   0122_16 = 160K entries, smallest size, lowest accuracy (set cld2_table_size=0)
 #   0122_19 = 192K entries, medium size, medium accuracy (set cld2_table_size=1)
 #   0122_2  = 256K entries, largest size, highest accuracy (set cld2_table_size=2)
 #
 # For the CJK bigram file (cld_generated_cjk_delta_bi*.cc), always use
 # cld_generated_cjk_delta_bi_4.cc, as this is intended for use with Chromium.
 # The _32 variant of the file is intended for applications that use the full
 # 175-language version of CLD2.
 
 {
   'targets': [
     {
-      'target_name': 'cld_2',
-      'type': 'static_library',
+      'target_name': 'cld2_dynamic_data_tool',
+      'type': 'executable',
       'include_dirs': [
         'src/internal',
         'src/public',
       ],
       'sources': [
+        'src/internal/cld2_dynamic_data.h',

[palmer, 2014/03/24 19:01:47]

How is this code getting tested? Is there a fuzzer? Et c.

[Andrew Hayden (chromium.org), 2014/03/26 15:34:11]

The testing is done via unit tests in CLD2, which were modified to fully
exercise the dynamic data use cases. All unit tests in CLD2 can be built in
either dynamic or static mode. Of course, they all pass.

There's no fuzzer. My stance so far has been that...
1. On Android, this file will be part of a signed APK that we generated and that
lives in the application's private directory. Therefore it is secure, and
shouldn't be any more of a security concern than any other data files we might
have.
2. On other platforms, the file is as well protected as the OS makes it, neither
more or less secure than anything else Chromium deals with.

It's not abundantly clear to me what a fuzzer would be able to do here, since
the data within the frozen structures is all critical to the correct functioning
of the code (e.g., shifts and offsets in arrays of characters). Fuzzing anything
in the data set would immediately and completely break the code in weird ways.
IANAE on this, but if you have docs on chromium fuzzers that I could somehow
adapt I'd be happy to take a look.

I guess my point is that right now I think we're as (in)secure as we are with
non-dynamic CLD1; anyone able to access the filesystem can corrupt the CLD data,
be that within the readonly block in the .so or within the standalone data file.
CLD has not and does not have any hardening against attacks of this type.

+        'src/internal/cld2_dynamic_data.cc',
+        'src/internal/cld2_dynamic_data_extractor.h',
+        'src/internal/cld2_dynamic_data_extractor.cc',
+        'src/internal/cld2_dynamic_data_loader.h',
+        'src/internal/cld2_dynamic_data_loader.cc',
+        'src/internal/cld2_dynamic_data_tool.cc',
         'src/internal/cld2_generated_cjk_compatible.cc',
         'src/internal/cld2_generated_deltaoctachrome0122.cc',
         'src/internal/cld2_generated_distinctoctachrome0122.cc',
         'src/internal/cld2tablesummary.h',
         'src/internal/cld_generated_cjk_delta_bi_4.cc',
         'src/internal/cld_generated_cjk_uni_prop_80.cc',
         'src/internal/cld_generated_score_quad_octa_0122_2.cc',
         'src/internal/cldutil.cc',
         'src/internal/cldutil.h',
         'src/internal/cldutil_shared.cc',
@@ skipping 52 matching lines @@
            ],
         }],
         ['cld2_table_size==2', {
           # Large table sizes
           'sources+': [
             'src/internal/cld2_generated_quadchrome0122_2.cc',
            ],
         }],
       ],
     },
+    {
+      'target_name': 'cld_2',
+      'type': 'static_library',
+      'include_dirs': [
+        'src/internal',
+        'src/public',
+      ],
+      'sources': [
+        'src/internal/cld2tablesummary.h',
+        'src/internal/cldutil.cc',
+        'src/internal/cldutil.h',
+        'src/internal/cldutil_shared.cc',
+        'src/internal/cldutil_shared.h',
+        'src/internal/compact_lang_det.cc',
+        'src/internal/compact_lang_det_hint_code.cc',
+        'src/internal/compact_lang_det_hint_code.h',
+        'src/internal/compact_lang_det_impl.cc',
+        'src/internal/compact_lang_det_impl.h',
+        'src/internal/debug.h',
+        'src/internal/debug_empty.cc',
+        'src/internal/fixunicodevalue.cc',
+        'src/internal/fixunicodevalue.h',
+        'src/internal/generated_distinct_bi_0.cc',
+        'src/internal/generated_entities.cc',
+        'src/internal/generated_language.cc',
+        'src/internal/generated_language.h',
+        'src/internal/generated_ulscript.cc',
+        'src/internal/generated_ulscript.h',
+        'src/internal/getonescriptspan.cc',
+        'src/internal/getonescriptspan.h',
+        'src/internal/integral_types.h',
+        'src/internal/lang_script.cc',
+        'src/internal/lang_script.h',
+        'src/internal/langspan.h',
+        'src/internal/offsetmap.cc',
+        'src/internal/offsetmap.h',
+        'src/internal/port.h',
+        'src/internal/scoreonescriptspan.cc',
+        'src/internal/scoreonescriptspan.h',
+        'src/internal/stringpiece.h',
+        'src/internal/tote.cc',
+        'src/internal/tote.h',
+        'src/internal/utf8prop_lettermarkscriptnum.h',
+        'src/internal/utf8repl_lettermarklower.h',
+        'src/internal/utf8scannot_lettermarkspecial.h',
+        'src/internal/utf8statetable.cc',
+        'src/internal/utf8statetable.h',
+        'src/public/compact_lang_det.h',
+        'src/public/encodings.h',
+      ],
+      'conditions': [
+        ['OS=="win"', {
+          'msvs_disabled_warnings': [4267],
+        }],
+        # Dynamic or vanilla mode
+        ['cld_dynamic==1',
+          {
+            # Enable dynamic mode, which allows CLD to read its data from a file
+            # instead of requiring the data to be linked into the library.
+            # You must generate the data file and check it in using the
+            # cld2_dynamic_data_tool.
+            # build/common.gypi takes care of setting -D CLD2_MODE_DYNAMIC
+            'sources+': [
+              'src/internal/cld2_dynamic_data.h',
+              'src/internal/cld2_dynamic_data.cc',
+              'src/internal/cld2_dynamic_data_loader.h',
+              'src/internal/cld2_dynamic_data_loader.cc',
+            ],
+          },
+          {
+            # Else it's regular mode, so link in all the data blobs directly.
+            'sources+': [
+              'src/internal/cld2_generated_cjk_compatible.cc',
+              'src/internal/cld2_generated_deltaoctachrome0122.cc',
+              'src/internal/cld2_generated_distinctoctachrome0122.cc',
+              'src/internal/cld_generated_cjk_delta_bi_4.cc',
+              'src/internal/cld_generated_cjk_uni_prop_80.cc',
+              'src/internal/cld_generated_score_quad_octa_0122_2.cc',
+              'src/internal/generated_distinct_bi_0.cc',
+            ],
+            'conditions': [
+              ['cld2_table_size==0', {
+                # Small table sizes
+                'sources+': [
+                  'src/internal/cld2_generated_quadchrome0122_16.cc',
+                 ],
+              }],
+              ['cld2_table_size==1', {
+                # Medium table sizes
+                'sources+': [
+                  'src/internal/cld2_generated_quadchrome0122_19.cc',
+                 ],
+              }],
+              ['cld2_table_size==2', {
+                # Large table sizes
+                'sources+': [
+                  'src/internal/cld2_generated_quadchrome0122_2.cc',
+                 ],
+              }],
+            ],
+          },
+        ],
+      ],
+    },
   ],
 }