Framework for iterating over certificates in CT database from Chromium code. (not for review)

***** MOVED TO: https://chromium-review.googlesource.com/580490 ******

Framework for iterating over certificates in CT database from Chromium code. (not for review).

See the README file for details on how to use this.

[eroman, 2015-10-24 00:51:32]

Description was changed from

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)

This is not for review, and uploaded for my own backup purposes.

Basically just some code wrappers for iterating over the Certificate
Transparency certificates locally for experimentation/testing purposes.

(The database has about 8 million certificates, and takes 11GB on disk).
==========

to

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)

 ===================
 One time setup
 ===================
 
 ---------------------------
 1. Apply this patch
 ---------------------------
 
 $ git new-branch CERT_MAPPER
 $ git cl patch 1238413004
 
 ---------------------------
 2. Download CT database
 ---------------------------
 
 If you haven't already, you need to download the full CT database. This takes a
 long time as it is really big (22GB). You can speed things up if you ask
 someone to copy it over for you instead.
 
 $ cd src
 $ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db
 
 ---------------------------
 3. Dump all the certificates from CT database
 ---------------------------
 
 Rather than operate directly on the CT database, the cert mapper iterates over
 a simpler (and smaller) file, which just contains the DER for each certificate.
 This file is about half the size of the CT.db.
 
 $ cd src
 $ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin
 
 This will take a few minutes, after which CERTS.bin will have been written.
 This is the input file that you pass to the cert_mapper.
 
 ======================================================
 How to run code over the certificates in CT database
 ======================================================
 
 Add your code into:
   net/tools/cert_mapper/my_visitor.cc
 
 You can run the default mapper by running cert_mapper:
 
  $ cd src
  $ ninja -C out/Release cert_mapper
  $ ./out/Release/cert_mapper  --input=CT.db 
 
 NOTES:
   * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
     re-generate the ninja files with "gn gen out/Release"
 
   * Because --input=PATH is all one shell token, things like '~' won't get
     expanded. If it is not finding the file try giving an absolute path
     instead.
 
 When cert_mapper runs it will output something like this:
 
 -------------------------------
 Aggregate bucket counts
 -------------------------------
    b0     7966695    100.000%     ParseCertificate() [OK]
    b1         775      0.010%     ParseTbsCertificate() [FAIL]
    b2     7965920     99.990%     ParseTbsCertificate() [OK]
 
 
 Meaning of the columns:
 
  First column (stuff like "b0"):
    These are identifiers for the buckets. It is only really relevant to help
    you find the corresponding output file that contain samples for the bucket
    (will be named things like b0.pem). To save the sample files, you need to
    specify the --output flag.
 
 Second column (stuff like 775):
    This is the count of certificates that were placed into the bucket.
 
 Third column (stuff like 99.990%):
    This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
    percentages may add up to more than 100%
==========

[eroman, 2015-10-24 00:52:00]

Description was changed from

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)

 ===================
 One time setup
 ===================
 
 ---------------------------
 1. Apply this patch
 ---------------------------
 
 $ git new-branch CERT_MAPPER
 $ git cl patch 1238413004
 
 ---------------------------
 2. Download CT database
 ---------------------------
 
 If you haven't already, you need to download the full CT database. This takes a
 long time as it is really big (22GB). You can speed things up if you ask
 someone to copy it over for you instead.
 
 $ cd src
 $ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db
 
 ---------------------------
 3. Dump all the certificates from CT database
 ---------------------------
 
 Rather than operate directly on the CT database, the cert mapper iterates over
 a simpler (and smaller) file, which just contains the DER for each certificate.
 This file is about half the size of the CT.db.
 
 $ cd src
 $ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin
 
 This will take a few minutes, after which CERTS.bin will have been written.
 This is the input file that you pass to the cert_mapper.
 
 ======================================================
 How to run code over the certificates in CT database
 ======================================================
 
 Add your code into:
   net/tools/cert_mapper/my_visitor.cc
 
 You can run the default mapper by running cert_mapper:
 
  $ cd src
  $ ninja -C out/Release cert_mapper
  $ ./out/Release/cert_mapper  --input=CT.db 
 
 NOTES:
   * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
     re-generate the ninja files with "gn gen out/Release"
 
   * Because --input=PATH is all one shell token, things like '~' won't get
     expanded. If it is not finding the file try giving an absolute path
     instead.
 
 When cert_mapper runs it will output something like this:
 
 -------------------------------
 Aggregate bucket counts
 -------------------------------
    b0     7966695    100.000%     ParseCertificate() [OK]
    b1         775      0.010%     ParseTbsCertificate() [FAIL]
    b2     7965920     99.990%     ParseTbsCertificate() [OK]
 
 
 Meaning of the columns:
 
  First column (stuff like "b0"):
    These are identifiers for the buckets. It is only really relevant to help
    you find the corresponding output file that contain samples for the bucket
    (will be named things like b0.pem). To save the sample files, you need to
    specify the --output flag.
 
 Second column (stuff like 775):
    This is the count of certificates that were placed into the bucket.
 
 Third column (stuff like 99.990%):
    This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
    percentages may add up to more than 100%
==========

to

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)
===================
One time setup
===================

---------------------------
1. Apply this patch
---------------------------

$ git new-branch CERT_MAPPER
$ git cl patch 1238413004

---------------------------
2. Download CT database
---------------------------

If you haven't already, you need to download the full CT database. This takes a
long time as it is really big (22GB). You can speed things up if you ask
someone to copy it over for you instead.

$ cd src
$ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db

---------------------------
3. Dump all the certificates from CT database
---------------------------

Rather than operate directly on the CT database, the cert mapper iterates over
a simpler (and smaller) file, which just contains the DER for each certificate.
This file is about half the size of the CT.db.

$ cd src
$ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin

This will take a few minutes, after which CERTS.bin will have been written.
This is the input file that you pass to the cert_mapper.

======================================================
How to run code over the certificates in CT database
======================================================

Add your code into:
  net/tools/cert_mapper/my_visitor.cc

You can run the default mapper by running cert_mapper:

 $ cd src
 $ ninja -C out/Release cert_mapper
 $ ./out/Release/cert_mapper  --input=CT.db 

NOTES:
  * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
    re-generate the ninja files with "gn gen out/Release"

  * Because --input=PATH is all one shell token, things like '~' won't get
    expanded. If it is not finding the file try giving an absolute path
    instead.

When cert_mapper runs it will output something like this:

-------------------------------
Aggregate bucket counts
-------------------------------
   b0     7966695    100.000%     ParseCertificate() [OK]
   b1         775      0.010%     ParseTbsCertificate() [FAIL]
   b2     7965920     99.990%     ParseTbsCertificate() [OK]


Meaning of the columns:

 First column (stuff like "b0"):
   These are identifiers for the buckets. It is only really relevant to help
   you find the corresponding output file that contain samples for the bucket
   (will be named things like b0.pem). To save the sample files, you need to
   specify the --output flag.

Second column (stuff like 775):
   This is the count of certificates that were placed into the bucket.

Third column (stuff like 99.990%):
   This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
   percentages may add up to more than 100%
==========

[eroman, 2015-10-24 00:52:08]

Description was changed from

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)
===================
One time setup
===================

---------------------------
1. Apply this patch
---------------------------

$ git new-branch CERT_MAPPER
$ git cl patch 1238413004

---------------------------
2. Download CT database
---------------------------

If you haven't already, you need to download the full CT database. This takes a
long time as it is really big (22GB). You can speed things up if you ask
someone to copy it over for you instead.

$ cd src
$ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db

---------------------------
3. Dump all the certificates from CT database
---------------------------

Rather than operate directly on the CT database, the cert mapper iterates over
a simpler (and smaller) file, which just contains the DER for each certificate.
This file is about half the size of the CT.db.

$ cd src
$ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin

This will take a few minutes, after which CERTS.bin will have been written.
This is the input file that you pass to the cert_mapper.

======================================================
How to run code over the certificates in CT database
======================================================

Add your code into:
  net/tools/cert_mapper/my_visitor.cc

You can run the default mapper by running cert_mapper:

 $ cd src
 $ ninja -C out/Release cert_mapper
 $ ./out/Release/cert_mapper  --input=CT.db 

NOTES:
  * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
    re-generate the ninja files with "gn gen out/Release"

  * Because --input=PATH is all one shell token, things like '~' won't get
    expanded. If it is not finding the file try giving an absolute path
    instead.

When cert_mapper runs it will output something like this:

-------------------------------
Aggregate bucket counts
-------------------------------
   b0     7966695    100.000%     ParseCertificate() [OK]
   b1         775      0.010%     ParseTbsCertificate() [FAIL]
   b2     7965920     99.990%     ParseTbsCertificate() [OK]


Meaning of the columns:

 First column (stuff like "b0"):
   These are identifiers for the buckets. It is only really relevant to help
   you find the corresponding output file that contain samples for the bucket
   (will be named things like b0.pem). To save the sample files, you need to
   specify the --output flag.

Second column (stuff like 775):
   This is the count of certificates that were placed into the bucket.

Third column (stuff like 99.990%):
   This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
   percentages may add up to more than 100%
==========

to

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)

===================
One time setup
===================

---------------------------
1. Apply this patch
---------------------------

$ git new-branch CERT_MAPPER
$ git cl patch 1238413004

---------------------------
2. Download CT database
---------------------------

If you haven't already, you need to download the full CT database. This takes a
long time as it is really big (22GB). You can speed things up if you ask
someone to copy it over for you instead.

$ cd src
$ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db

---------------------------
3. Dump all the certificates from CT database
---------------------------

Rather than operate directly on the CT database, the cert mapper iterates over
a simpler (and smaller) file, which just contains the DER for each certificate.
This file is about half the size of the CT.db.

$ cd src
$ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin

This will take a few minutes, after which CERTS.bin will have been written.
This is the input file that you pass to the cert_mapper.

======================================================
How to run code over the certificates in CT database
======================================================

Add your code into:
  net/tools/cert_mapper/my_visitor.cc

You can run the default mapper by running cert_mapper:

 $ cd src
 $ ninja -C out/Release cert_mapper
 $ ./out/Release/cert_mapper  --input=CT.db 

NOTES:
  * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
    re-generate the ninja files with "gn gen out/Release"

  * Because --input=PATH is all one shell token, things like '~' won't get
    expanded. If it is not finding the file try giving an absolute path
    instead.

When cert_mapper runs it will output something like this:

-------------------------------
Aggregate bucket counts
-------------------------------
   b0     7966695    100.000%     ParseCertificate() [OK]
   b1         775      0.010%     ParseTbsCertificate() [FAIL]
   b2     7965920     99.990%     ParseTbsCertificate() [OK]


Meaning of the columns:

 First column (stuff like "b0"):
   These are identifiers for the buckets. It is only really relevant to help
   you find the corresponding output file that contain samples for the bucket
   (will be named things like b0.pem). To save the sample files, you need to
   specify the --output flag.

Second column (stuff like 775):
   This is the count of certificates that were placed into the bucket.

Third column (stuff like 99.990%):
   This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
   percentages may add up to more than 100%
==========

[eroman, 2015-10-24 01:20:28]

Description was changed from

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)

===================
One time setup
===================

---------------------------
1. Apply this patch
---------------------------

$ git new-branch CERT_MAPPER
$ git cl patch 1238413004

---------------------------
2. Download CT database
---------------------------

If you haven't already, you need to download the full CT database. This takes a
long time as it is really big (22GB). You can speed things up if you ask
someone to copy it over for you instead.

$ cd src
$ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db

---------------------------
3. Dump all the certificates from CT database
---------------------------

Rather than operate directly on the CT database, the cert mapper iterates over
a simpler (and smaller) file, which just contains the DER for each certificate.
This file is about half the size of the CT.db.

$ cd src
$ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin

This will take a few minutes, after which CERTS.bin will have been written.
This is the input file that you pass to the cert_mapper.

======================================================
How to run code over the certificates in CT database
======================================================

Add your code into:
  net/tools/cert_mapper/my_visitor.cc

You can run the default mapper by running cert_mapper:

 $ cd src
 $ ninja -C out/Release cert_mapper
 $ ./out/Release/cert_mapper  --input=CT.db 

NOTES:
  * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
    re-generate the ninja files with "gn gen out/Release"

  * Because --input=PATH is all one shell token, things like '~' won't get
    expanded. If it is not finding the file try giving an absolute path
    instead.

When cert_mapper runs it will output something like this:

-------------------------------
Aggregate bucket counts
-------------------------------
   b0     7966695    100.000%     ParseCertificate() [OK]
   b1         775      0.010%     ParseTbsCertificate() [FAIL]
   b2     7965920     99.990%     ParseTbsCertificate() [OK]


Meaning of the columns:

 First column (stuff like "b0"):
   These are identifiers for the buckets. It is only really relevant to help
   you find the corresponding output file that contain samples for the bucket
   (will be named things like b0.pem). To save the sample files, you need to
   specify the --output flag.

Second column (stuff like 775):
   This is the count of certificates that were placed into the bucket.

Third column (stuff like 99.990%):
   This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
   percentages may add up to more than 100%
==========

to

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)

===================
One time setup
===================

---------------------------
1. Apply this patch
---------------------------

$ git new-branch CERT_MAPPER
$ git cl patch 1238413004

---------------------------
2. Download CT database
---------------------------

If you haven't already, you need to download the full CT database. This takes a
long time (order of hours), and will save a file about 22GB large. (If you know
someone that has already downloaded this, you may want to copy it from them
instead of fetching it yourself.)

$ cd src
$ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db

---------------------------
3. Dump all the certificates from CT database
---------------------------

Rather than operate directly on the CT database, the cert mapper iterates over
a simpler (and smaller) file, which just contains the DER for each certificate.
This file is about half the size of the CT.db (making it around 11GB).

$ cd src
$ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin

This will take about 10 minutes to run, after which CERTS.bin will have been
written. This is the input file that you pass to the cert_mapper.

======================================================
How to run code over the certificates in CT database (or a PEM file)
======================================================

Add your code into:
  net/tools/cert_mapper/my_visitor.cc

And then execute:

 $ cd src
 $ ninja -C out/Release cert_mapper
 $ ./out/Release/cert_mapper  --input=CERTS.bin

NOTES:
  * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
    re-generate the ninja files with "gn gen out/Release"

  * The extension for --input MATTERS. Either use a file with .bin extension,
    or .pem. .bin is the internal (efficient) file format, whereas .pem is a
    regular PEM file containing CERTIFICATES. For instance you can feed back the
    samples files (which are .pem).

  * Because --input=PATH is all one shell token, things like '~' won't get
    expanded. If it is not finding the file try giving an absolute path
    instead.

======================================================
Meaning of cert_mapper's output
======================================================

When cert_mapper runs it will output something like this:

-------------------------------
Aggregate bucket counts
-------------------------------
   b0     7966695    100.000%     ParseCertificate() [OK]
   b1         775      0.010%     ParseTbsCertificate() [FAIL]
   b2     7965920     99.990%     ParseTbsCertificate() [OK]

Meaning of the columns:

 First column (stuff like "b0"):
   These are identifiers for the buckets. It is only really relevant to help
   you find the corresponding output file that contain samples for the bucket
   (will be named things like b0.pem). To save the sample files, you need to
   specify the --output flag. For instance in this example you could open up
   out_dir/b1.pem to see certificates which failed parsing for TBSCertificate.

Second column (stuff like 775):
   This is the count of certificates that were placed into the bucket.

Third column (stuff like 99.990%):
   This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
   percentages may add up to more than 100%
==========

[eroman, 2015-10-28 20:53:57]

Description was changed from

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review)

===================
One time setup
===================

---------------------------
1. Apply this patch
---------------------------

$ git new-branch CERT_MAPPER
$ git cl patch 1238413004

---------------------------
2. Download CT database
---------------------------

If you haven't already, you need to download the full CT database. This takes a
long time (order of hours), and will save a file about 22GB large. (If you know
someone that has already downloaded this, you may want to copy it from them
instead of fetching it yourself.)

$ cd src
$ ./net/tools/cert_mapper/sync-ct-database.sh  CT.db

---------------------------
3. Dump all the certificates from CT database
---------------------------

Rather than operate directly on the CT database, the cert mapper iterates over
a simpler (and smaller) file, which just contains the DER for each certificate.
This file is about half the size of the CT.db (making it around 11GB).

$ cd src
$ ./net/tools/cert_mapper/dump-certs.sh  CT.db   CERTS.bin

This will take about 10 minutes to run, after which CERTS.bin will have been
written. This is the input file that you pass to the cert_mapper.

======================================================
How to run code over the certificates in CT database (or a PEM file)
======================================================

Add your code into:
  net/tools/cert_mapper/my_visitor.cc

And then execute:

 $ cd src
 $ ninja -C out/Release cert_mapper
 $ ./out/Release/cert_mapper  --input=CERTS.bin

NOTES:
  * If you get "ninja: error: unknown target 'cert_mapper'" then you need to
    re-generate the ninja files with "gn gen out/Release"

  * The extension for --input MATTERS. Either use a file with .bin extension,
    or .pem. .bin is the internal (efficient) file format, whereas .pem is a
    regular PEM file containing CERTIFICATES. For instance you can feed back the
    samples files (which are .pem).

  * Because --input=PATH is all one shell token, things like '~' won't get
    expanded. If it is not finding the file try giving an absolute path
    instead.

======================================================
Meaning of cert_mapper's output
======================================================

When cert_mapper runs it will output something like this:

-------------------------------
Aggregate bucket counts
-------------------------------
   b0     7966695    100.000%     ParseCertificate() [OK]
   b1         775      0.010%     ParseTbsCertificate() [FAIL]
   b2     7965920     99.990%     ParseTbsCertificate() [OK]

Meaning of the columns:

 First column (stuff like "b0"):
   These are identifiers for the buckets. It is only really relevant to help
   you find the corresponding output file that contain samples for the bucket
   (will be named things like b0.pem). To save the sample files, you need to
   specify the --output flag. For instance in this example you could open up
   out_dir/b1.pem to see certificates which failed parsing for TBSCertificate.

Second column (stuff like 775):
   This is the count of certificates that were placed into the bucket.

Third column (stuff like 99.990%):
   This is the count / TOTAL_CERTIFICATES_ITERATED_OVER. As a result, the
   percentages may add up to more than 100%
==========

to

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review).

See the README file for information on how to use this.
==========

[eroman, 2015-10-28 20:54:33]

Description was changed from

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review).

See the README file for information on how to use this.
==========

to

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review).

See the README file for details on how to use this.
==========

[eroman, 2017-07-20 20:56:04]

Description was changed from

==========
Framework for iterating over certificates in CT database from Chromium code.
(not for review).

See the README file for details on how to use this.
==========

to

==========
***** MOVED TO: https://chromium-review.googlesource.com/580490 ******

Framework for iterating over certificates in CT database from Chromium code.
(not for review).

See the README file for details on how to use this.
==========