Add scripts for distillability modelling

Add scripts for distillability modeling

The model produced from these scripts was added to Chrome here:
http://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/

Highlight of improvements:
* Feature extraction
  - Add new features that are effective and also cheap to compute in
    Blink
  - Remove expensive features from the derived features
* Scrawling
  - Merge feature extraction and screenshot taking scripts
  - Enable concurrent scrawling for parallelization
  - Add MHTML archiving for reproducibility
* Labeling server
  - Automatically advance to the next unrated item
  - Prefetch screenshots of next item and next unrated item
  - Use location.hash to access individual entires. If not specified,
    randomly pick one.
  - Restore state from all the archive files
  - Fix HTTP expiration of files
  - Improve color scheme, show decoded URLs, show rating stats, etc
* Add documentation

R=mdjones@chromium.org

Committed: 72998ef3164652dd7aeab7e211dfe100dcd46378

[wychen, 2016-01-26 02:27:38]

Patchset #2 (id:20001) has been deleted

[wychen, 2016-01-26 02:28:40]

Description was changed from

==========
machine learning pipeline for distillability (draft)
==========

to

==========
Add scripts for distillability modelling
==========

[wychen, 2016-01-26 02:28:52]

wychen@chromium.org changed reviewers:
+ mdjones@chromium.org

[wychen, 2016-01-26 02:28:53]

PTAL

[mdjones, 2016-02-03 16:53:36]

Did a quick scan and lgtm. At some point I would add documentation for some of
the function in the code though.

[wychen, 2016-02-10 00:34:13]

Description was changed from

==========
Add scripts for distillability modelling
==========

to

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804
==========

[wychen, 2016-02-10 01:53:58]

Description was changed from

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804
==========

to

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on http://crrev.com/1289123002/.
==========

[wychen, 2016-02-10 01:54:27]

Description was changed from

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on http://crrev.com/1289123002/.
==========

to

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/.
==========

[wychen, 2016-02-10 01:54:40]

Description was changed from

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/.
==========

to

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman@'s work here:
http://crrev.com/1289123002/.
==========

[wychen, 2016-02-10 01:55:02]

Description was changed from

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman@'s work here:
http://crrev.com/1289123002/.
==========

to

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/.
==========

[wychen, 2016-02-10 20:13:57]

Description was changed from

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/.
==========

to

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/
==========

[wychen, 2016-02-11 04:29:18]

Description was changed from

==========
Add scripts for distillability modelling

The model produced from these scripts were added to Chrome here:
https://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/
==========

to

==========
Add scripts for distillability modeling

The model produced from these scripts was added to Chrome here:
http://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/

Highlight of improvements:
* Feature extraction
  - Add new features that are effective and also cheap to compute in
    Blink
  - Remove expensive features from the derived features
* Scrawling
  - Merge feature extraction and screenshot taking scripts
  - Enable concurrent scrawling for parallelization
  - Add MHTML archiving for reproducibility
* Labeling server
  - Automatically advance to the next unrated item
  - Prefetch screenshots of next item and next unrated item
  - Use location.hash to access individual entires. If not specified,
    randomly pick one.
  - Restore state from all the archive files
  - Fix HTTP expiration of files
  - Improve color scheme, show decoded URLs, show rating stats, etc
* Add documentation
==========

[wychen, 2016-02-11 04:38:43]

Description was changed from

==========
Add scripts for distillability modeling

The model produced from these scripts was added to Chrome here:
http://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/

Highlight of improvements:
* Feature extraction
  - Add new features that are effective and also cheap to compute in
    Blink
  - Remove expensive features from the derived features
* Scrawling
  - Merge feature extraction and screenshot taking scripts
  - Enable concurrent scrawling for parallelization
  - Add MHTML archiving for reproducibility
* Labeling server
  - Automatically advance to the next unrated item
  - Prefetch screenshots of next item and next unrated item
  - Use location.hash to access individual entires. If not specified,
    randomly pick one.
  - Restore state from all the archive files
  - Fix HTTP expiration of files
  - Improve color scheme, show decoded URLs, show rating stats, etc
* Add documentation
==========

to

==========
Add scripts for distillability modeling

The model produced from these scripts was added to Chrome here:
http://crrev.com/8587df371e24c3003b743a3bb0fc87ad77152804

This is based on cjhopman's work here:
http://crrev.com/1289123002/

Highlight of improvements:
* Feature extraction
  - Add new features that are effective and also cheap to compute in
    Blink
  - Remove expensive features from the derived features
* Scrawling
  - Merge feature extraction and screenshot taking scripts
  - Enable concurrent scrawling for parallelization
  - Add MHTML archiving for reproducibility
* Labeling server
  - Automatically advance to the next unrated item
  - Prefetch screenshots of next item and next unrated item
  - Use location.hash to access individual entires. If not specified,
    randomly pick one.
  - Restore state from all the archive files
  - Fix HTTP expiration of files
  - Improve color scheme, show decoded URLs, show rating stats, etc
* Add documentation

R=mdjones@chromium.org

Committed: 72998ef3164652dd7aeab7e211dfe100dcd46378
==========

[wychen, 2016-02-11 04:38:45]

Committed patchset #3 (id:60001) manually as
72998ef3164652dd7aeab7e211dfe100dcd46378 (presubmit successful).