| OLD | NEW |
| 1 # DOM Distiller | 1 # DOM Distiller |
| 2 | 2 |
| 3 DOM Distiller aims to provide a better reading experience by distilling the | 3 DOM Distiller aims to provide a better reading experience by distilling the |
| 4 content of the page. This distilled content can then be used in a variety of | 4 content of the page. This distilled content can then be used in a variety of |
| 5 ways. | 5 ways. |
| 6 | 6 |
| 7 The current efforts that will be powered by DOM Distiller: | 7 The current efforts that will be powered by DOM Distiller: |
| 8 - Reader mode: a mobile-friendly viewing mode for Chrome mobile | 8 |
| 9 - Reader mode: a mobile-friendly viewing mode for Chrome mobile |
| 9 | 10 |
| 10 ## How to use Reader mode on mobile Chrome | 11 ## How to use Reader mode on mobile Chrome |
| 11 | 12 |
| 12 - Open Chrome on your mobile device | 13 - Open Chrome on your mobile device |
| 13 - Navigate to [chrome://flags](chrome://flags) and search for "Reader mode" | 14 - Navigate to [chrome://flags](chrome://flags) and search for "Reader mode" |
| 14 (Menu -> Find in page -> Enable Reader Mode Toolbar Icon), or directly go to | 15 (Menu -> Find in page -> Enable Reader Mode Toolbar Icon), or directly go to |
| 15 [chrome://flags#enable-reader-mode-toolbar-icon](chrome://flags#enable-reader-
mode-toolbar-icon) | 16 [chrome://flags#enable-reader-mode-toolbar-icon] |
| 16 - Click Enable to turn on Reader mode | 17 (chrome://flags#enable-reader-mode-toolbar-icon) |
| 17 - Click "Relaunch Now" at the bottom of the page | 18 - Click Enable to turn on Reader mode |
| 18 - Next time you're trying to read a page, tap on the "Reader mode" icon in | 19 - Click "Relaunch Now" at the bottom of the page |
| 19 the toolbar to try it out! | 20 - Next time you're trying to read a page, tap on the "Reader mode" icon in the |
| 21 toolbar to try it out! |
| 20 | 22 |
| 21 # Continuous integration | 23 # Continuous integration |
| 22 | 24 |
| 23 - [](https://drone.io/github.com/chromium/dom-distiller/latest) | 25 - [![Build Status] |
| 24 - [drone.io waterfall](https://drone.io/github.com/chromium/dom-distiller) | 26 (https://drone.io/github.com/chromium/dom-distiller/status.png)] |
| 27 (https://drone.io/github.com/chromium/dom-distiller/latest) |
| 28 - [drone.io waterfall](https://drone.io/github.com/chromium/dom-distiller) |
| 25 | 29 |
| 26 # Get the code | 30 # Get the code |
| 27 | 31 |
| 28 In a folder where you want the code (outside of the chromium checkout): | 32 In a folder where you want the code (outside of the chromium checkout): |
| 29 | 33 |
| 30 ```bash | 34 ```bash |
| 31 git clone https://github.com/chromium/dom-distiller.git | 35 git clone https://github.com/chromium/dom-distiller.git |
| 32 ``` | 36 ``` |
| 37 |
| 33 A `dom-distiller` folder will be created in the folder you run that command. | 38 A `dom-distiller` folder will be created in the folder you run that command. |
| 34 | 39 |
| 35 # Environment setup | 40 # Environment setup |
| 36 | 41 |
| 37 Before you build for the first time, you need to install the build dependencies. | 42 Before you build for the first time, you need to install the build dependencies. |
| 38 | 43 |
| 39 For all platforms, it is require to download and install | 44 For all platforms, it is require to download and install [Google Chrome browser] |
| 40 [Google Chrome browser](https://www.google.com/chrome/browser/desktop/). | 45 (https://www.google.com/chrome/browser/desktop/). |
| 41 | 46 |
| 42 ChromeDriver requires Google Chrome to be installed at a specific location | 47 ChromeDriver requires Google Chrome to be installed at a specific location (the |
| 43 (the default location for the platform). See | 48 default location for the platform). See [ChromeDriver documentation] |
| 44 [ChromeDriver documentation](https://code.google.com/p/selenium/wiki/ChromeDrive
r) | 49 (https://code.google.com/p/selenium/wiki/ChromeDriver) for details. |
| 45 for details. | |
| 46 | 50 |
| 47 ## Developing on Ubuntu/Debian | 51 ## Developing on Ubuntu/Debian |
| 48 | 52 |
| 49 Install the dependencies by entering the `dom-distiller` folder and running: | 53 Install the dependencies by entering the `dom-distiller` folder and running: |
| 54 |
| 50 ```bash | 55 ```bash |
| 51 sudo ./install-build-deps.sh | 56 sudo ./install-build-deps.sh |
| 52 ``` | 57 ``` |
| 53 | 58 |
| 54 Ubuntu 14.04 64-bit is recommended. | 59 Ubuntu 14.04 64-bit is recommended. |
| 55 | 60 |
| 56 ## Developing on Mac OS X | 61 ## Developing on Mac OS X |
| 57 | 62 |
| 58 - Install JDK 7 using either your organizations software management tool, | 63 - Install JDK 7 using either your organizations software management tool, or |
| 59 or download it from | 64 download it from [Oracle] |
| 60 [Oracle](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-
1880260.html). | 65 (http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880
260.html). |
| 61 - Install [Homebrew](http://brew.sh/). | 66 - Install [Homebrew](http://brew.sh/). |
| 62 - Install `ant` and `python` using Homebrew: | 67 - Install `ant` and `python` using Homebrew: |
| 63 ```bash | 68 |
| 64 brew install ant python | 69 ```bash |
| 65 ``` | 70 brew install ant python |
| 66 - Since both the protocol buffer compiler and Python bindings are needed, | 71 ``` |
| 67 install the `protobuf` package with the `--with-python` command line parameter: | 72 |
| 68 ```bash | 73 - Since both the protocol buffer compiler and Python bindings are needed, |
| 69 brew install protobuf --with-python | 74 install the `protobuf` package with the `--with-python` command line |
| 70 ``` | 75 parameter: |
| 71 - Create a folder named `buildtools` inside your DOM Distiller checkout | 76 |
| 72 - Download ChromeDriver (chromedriver_mac32.zip) from the | 77 ```bash |
| 73 [Download page](https://sites.google.com/a/chromium.org/chromedriver/downloads) | 78 brew install protobuf --with-python |
| 74 - Unzip the `chromedriver_mac32.zip` and ensure the binary ends up in your | 79 ``` |
| 75 `buildtools` folder. | 80 |
| 76 - Install the PyPI package management tool `pip` by running: | 81 - Create a folder named `buildtools` inside your DOM Distiller checkout |
| 77 ``` | 82 |
| 78 sudo easy_install pip | 83 - Download ChromeDriver (chromedriver_mac32.zip) from the [Download page] |
| 79 ``` | 84 (https://sites.google.com/a/chromium.org/chromedriver/downloads) |
| 80 - Install `selenium` using `pip`: | 85 |
| 81 ``` | 86 - Unzip the `chromedriver_mac32.zip` and ensure the binary ends up in your |
| 82 pip install --user selenium | 87 `buildtools` folder. |
| 83 ``` | 88 |
| 89 - Install the PyPI package management tool `pip` by running: |
| 90 |
| 91 ```bash |
| 92 sudo easy_install pip |
| 93 ``` |
| 94 |
| 95 - Install `selenium` using `pip`: |
| 96 |
| 97 ```bash |
| 98 pip install --user selenium |
| 99 ``` |
| 84 | 100 |
| 85 For the rest of this guide, there are sometimes references to a tool called | 101 For the rest of this guide, there are sometimes references to a tool called |
| 86 `xvfb` and specifically when running shell commands using `xvfb-run`. When you | 102 `xvfb` and specifically when running shell commands using `xvfb-run`. When you |
| 87 develop using a Mac OS X, you can remove that part of the command. For example | 103 develop using a Mac OS X, you can remove that part of the command. For example |
| 88 `xvfb-run echo` would just become `echo`. | 104 `xvfb-run echo` would just become `echo`. |
| 89 | 105 |
| 90 ## Developing with Vagrant | 106 ## Developing with Vagrant |
| 91 | 107 |
| 92 This option could be useful if you want to develop on an unsupported system | 108 This option could be useful if you want to develop on an unsupported system like |
| 93 like Windows or Red Hat Linux. Even if you are on a supported system but | 109 Windows or Red Hat Linux. Even if you are on a supported system but would rather |
| 94 would rather not touch the system too much, Vagrant is a viable alternative. | 110 not touch the system too much, Vagrant is a viable alternative. |
| 95 | 111 |
| 96 The Vagrant VM is based on Ubuntu 14.04. | 112 The Vagrant VM is based on Ubuntu 14.04. |
| 97 | 113 |
| 98 - [Install Vagrant](http://www.vagrantup.com/downloads.html) on your system. | 114 - [Install Vagrant](http://www.vagrantup.com/downloads.html) on your system. |
| 99 Version 1.7.2 or higher is recommended. | 115 Version 1.7.2 or higher is recommended. |
| 100 - Launch the Vagrant VM instance | 116 - Launch the Vagrant VM instance |
| 101 ```bash | 117 |
| 102 vagrant up | 118 ```bash |
| 103 ``` | 119 vagrant up |
| 104 - SSH to the VM | 120 ``` |
| 105 ```bash | 121 |
| 106 vagrant ssh | 122 - SSH to the VM |
| 107 ``` | 123 |
| 124 ```bash |
| 125 vagrant ssh |
| 126 ``` |
| 108 | 127 |
| 109 ## Tools for contributing | 128 ## Tools for contributing |
| 110 | 129 |
| 111 The DOM Distiller project uses the Chromium tools for collaboration. For code | 130 The DOM Distiller project uses the Chromium tools for collaboration. For code |
| 112 reviews, the | 131 reviews, the [Chromium Rietveld code review tool] |
| 113 [Chromium Rietveld code review tool](https://codereview.chromium.org/) is used | 132 (https://codereview.chromium.org/) is used and the set of tools found in |
| 114 and the set of tools found in `depot_tools` is also required. | 133 `depot_tools` is also required. |
| 115 | 134 |
| 116 To get `depot_tools`, follow the guide at | 135 To get `depot_tools`, follow the guide at [Chrome infrastructure documentation |
| 117 [Chrome infrastructure documentation for depot_tools](http://commondatastorage.g
oogleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.
html#_setting_up). | 136 for depot_tools] |
| 137 (http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs
/html/depot_tools_tutorial.html#_setting_up). |
| 118 | 138 |
| 119 The TL;DR of that is to run this from a folder where you install developer | 139 The TL;DR of that is to run this from a folder where you install developer |
| 120 tools, for example in your `$HOME` folder: | 140 tools, for example in your `$HOME` folder: |
| 141 |
| 121 ```bash | 142 ```bash |
| 122 git clone https://chromium.googlesource.com/chromium/tools/depot_tools | 143 git clone https://chromium.googlesource.com/chromium/tools/depot_tools |
| 123 export PATH="/path/to/depot_tools:$PATH" | 144 export PATH="/path/to/depot_tools:$PATH" |
| 124 ``` | 145 ``` |
| 125 | 146 |
| 126 You must also setup your local checkout needs to point to the Chromium Rietveld | 147 You must also setup your local checkout needs to point to the Chromium Rietveld |
| 127 server. This is a one-time setup for your checkout, so from your `dom-distiller` | 148 server. This is a one-time setup for your checkout, so from your `dom-distiller` |
| 128 checkout folder, run: | 149 checkout folder, run: |
| 150 |
| 129 ```bash | 151 ```bash |
| 130 git cl config | 152 git cl config |
| 131 ``` | 153 ``` |
| 132 - `Rietveld server`: `https://codereview.chromium.org` | 154 |
| 133 - You can leave the rest of the fields blank. | 155 - `Rietveld server`: `https://codereview.chromium.org` |
| 156 - You can leave the rest of the fields blank. |
| 134 | 157 |
| 135 # Building | 158 # Building |
| 136 | 159 |
| 137 ## Using ant | 160 ## Using ant |
| 138 | 161 |
| 139 `ant` is the tool we use to build, and the available targets can be listed using | 162 `ant` is the tool we use to build, and the available targets can be listed using |
| 140 `ant -p`, but the typical targets you might use when you work on this project | 163 `ant -p`, but the typical targets you might use when you work on this project |
| 141 is: | 164 is: |
| 142 | 165 |
| 143 - `ant test` Runs all tests. | 166 - `ant test` Runs all tests. |
| 144 - `ant test -Dtest.filter=$FILTER_PATTERN` where `$FILTER_PATTERN` is a [gtest\_
filter | 167 - `ant test -Dtest.filter=$FILTER_PATTERN` where `$FILTER_PATTERN` is a |
| 145 pattern] | 168 [gtest\_filter pattern] |
| 146 (https://code.google.com/p/googletest/wiki/AdvancedGuide#Running_a_Subset_of_the
_Tests). | 169 (https://code.google.com/p/googletest/wiki/AdvancedGuide#Running_a_Subset_of
_the_Tests). |
| 147 For example `*.FilterTest.*:*Foo*-*Bar*` would run all tests containing `.Filter
Test.` | 170 For example `*.FilterTest.*:*Foo*-*Bar*` would run all tests containing |
| 148 and `Foo`, but not those with `Bar`. | 171 `.FilterTest.` and `Foo`, but not those with `Bar`. |
| 149 - `ant gwtc` compiles .class+.java files to JavaScript. Standalone JavaScript | 172 - `ant gwtc` compiles .class+.java files to JavaScript. Standalone JavaScript |
| 150 is available at `war/domdistiller/domdistiller.nocache.js`. | 173 is available at `war/domdistiller/domdistiller.nocache.js`. |
| 151 - `ant gwtc.jstests` creates a standalone JavaScript for the tests. | 174 - `ant gwtc.jstests` creates a standalone JavaScript for the tests. |
| 152 - `ant package` Copies the main build artifacts into the `out/package` folder, | 175 - `ant package` Copies the main build artifacts into the `out/package` folder, |
| 153 typically the extracted JS and protocol buffer files. | 176 typically the extracted JS and protocol buffer files. |
| 154 | 177 |
| 155 # Contributing | 178 # Contributing |
| 156 | 179 |
| 157 You can use regular `git` command when developing in this project and use | 180 You can use regular `git` command when developing in this project and use `git |
| 158 `git cl` for collaboration. | 181 cl` for collaboration. |
| 159 | 182 |
| 160 ## Uploading a CL for review | 183 ## Uploading a CL for review |
| 161 | 184 |
| 162 On your branch, run: `git cl upload`. The first time you do this, you will have | 185 On your branch, run: `git cl upload`. The first time you do this, you will have |
| 163 to provide a username and password. | 186 to provide a username and password. |
| 164 - For username, use your @chromium.org. account. | 187 |
| 165 - For password, get it from | 188 - For username, use your @chromium.org. account. |
| 166 [GoogleCode.com settings page](https://code.google.com/hosting/settings) when | 189 - For password, get it from [GoogleCode.com settings page] |
| 167 logged into your @chromium.org account, and add the full | 190 (https://code.google.com/hosting/settings) when logged into your |
| 168 `machine code.google.com login` line to your `~/.netrc` file. | 191 @chromium.org account, and add the full `machine code.google.com login` line |
| 192 to your `~/.netrc` file. |
| 169 | 193 |
| 170 ## Committing a CL | 194 ## Committing a CL |
| 171 | 195 |
| 172 - Change upstream to remote master, push cl, then revert upstream to local: | 196 - Change upstream to remote master, push cl, then revert upstream to local: |
| 173 ```bash | 197 |
| 174 git branch -u origin/master | 198 ```bash |
| 175 git cl land | 199 git branch -u origin/master |
| 176 git branch -u master | 200 git cl land |
| 177 ``` | 201 git branch -u master |
| 178 - For username, use your GitHub account name (the username, not the full | 202 ``` |
| 179 e-mail). | 203 |
| 180 - For password, use your GitHub password. | 204 - For username, use your GitHub account name (the username, not the full |
| 181 - If you have two-factor authentication enabled, create a personal access | 205 e-mail). |
| 182 token at your | 206 |
| 183 [application settings page](https://github.com/settings/applications) and use | 207 - For password, use your GitHub password. |
| 184 that as your password. | 208 |
| 209 - If you have two-factor authentication enabled, create a personal access |
| 210 token at your [application settings page] |
| 211 (https://github.com/settings/applications) and use that as your |
| 212 password. |
| 185 | 213 |
| 186 ## Code formatting | 214 ## Code formatting |
| 187 | 215 |
| 188 Before uploading a CL it is recommended to run `git cl format`. However, this | 216 Before uploading a CL it is recommended to run `git cl format`. However, this |
| 189 requires adding symbolic links to your chromium checkout. | 217 requires adding symbolic links to your chromium checkout. |
| 190 | 218 |
| 191 Inside the `buildtools` folder of your checkout, add the following symbolic | 219 Inside the `buildtools` folder of your checkout, add the following symbolic |
| 192 links: | 220 links: |
| 193 - `clang_format`» → `/path/to/chromium/src/buildtools/clang_format/` | 221 |
| 194 - `linux64` ->» → `/path/to/chromium/src/buildtools/linux64/` (only for Lin
ux 64-bit platform) | 222 - `clang_format` → `/path/to/chromium/src/buildtools/clang_format/` |
| 195 - `mac` → `/path/to/chromium/mac/buildtools/linux64/` (only for Mac platfor
m) | 223 - `linux64` → `/path/to/chromium/src/buildtools/linux64/` (only for Linux |
| 224 64-bit platform) |
| 225 - `mac` → `/path/to/chromium/mac/buildtools/linux64/` (only for Mac |
| 226 platform) |
| 196 | 227 |
| 197 Doing this enables you to run the command `git cl format` to fix the formatting | 228 Doing this enables you to run the command `git cl format` to fix the formatting |
| 198 of your code. | 229 of your code. |
| 199 | 230 |
| 200 # Run in Chrome for desktop | 231 # Run in Chrome for desktop |
| 201 | 232 |
| 202 In this section, the following shell variables and are assumed correctly set: | 233 In this section, the following shell variables and are assumed correctly set: |
| 234 |
| 203 ```bash | 235 ```bash |
| 204 export CHROME_SRC=/path/to/chromium/src | 236 export CHROME_SRC=/path/to/chromium/src |
| 205 export DOM_DISTILLER_DIR=/path/to/dom-distiller | 237 export DOM_DISTILLER_DIR=/path/to/dom-distiller |
| 206 ``` | 238 ``` |
| 207 | 239 |
| 208 - Pull generated package (from ant package) into Chrome. You can use this handy | 240 - Pull generated package (from ant package) into Chrome. You can use this |
| 209 bash-function to help with that: | 241 handy bash-function to help with that: |
| 210 ```bash | 242 |
| 211 roll-distiller () { | 243 ```bash |
| 212 ( | 244 roll-distiller () { |
| 213 (cd $DOM_DISTILLER_DIR && ant package) && \ | 245 ( |
| 214 rm -rf $CHROME_SRC/third_party/dom_distiller_js/dist/* && \ | 246 (cd $DOM_DISTILLER_DIR && ant package) && \ |
| 215 cp -rf $DOM_DISTILLER_DIR/out/package/* $CHROME_SRC/third_party/dom_distille
r_js/dist/ && \ | 247 rm -rf $CHROME_SRC/third_party/dom_distiller_js/dist/* && \ |
| 216 touch $CHROME_SRC/components/resources/dom_distiller_resources.grdp | 248 cp -rf $DOM_DISTILLER_DIR/out/package/* $CHROME_SRC/third_party/dom_dist
iller_js/dist/ && \ |
| 217 ) | 249 touch $CHROME_SRC/components/resources/dom_distiller_resources.grdp |
| 218 } | 250 ) |
| 219 ``` | 251 } |
| 220 - From `$CHROME_SRC` run GYP to setup ninja build files using | 252 ``` |
| 221 ```bash | 253 |
| 222 build/gyp_chromium | 254 - From `$CHROME_SRC` run GYP to setup ninja build files using |
| 223 ``` | 255 |
| 256 ```bash |
| 257 build/gyp_chromium |
| 258 ``` |
| 224 | 259 |
| 225 ## Running the Chrome browser with distiller support | 260 ## Running the Chrome browser with distiller support |
| 226 | 261 |
| 227 - For running Chrome, you need to build the `chrome` target: | 262 - For running Chrome, you need to build the `chrome` target: |
| 228 ```bash | 263 |
| 229 ninja -C out/Debug chrome | 264 ```bash |
| 230 ``` | 265 ninja -C out/Debug chrome |
| 231 - Run chrome with DOM Distiller enabled: | 266 ``` |
| 232 ```bash | 267 |
| 233 out/Debug/chrome --enable-dom-distiller | 268 - Run chrome with DOM Distiller enabled: |
| 234 ``` | 269 |
| 235 - This adds a menu item `Distill page` that you can use to distill web pages. | 270 ```bash |
| 236 - You can also go to `chrome://dom-distiller` to access the debug page. | 271 out/Debug/chrome --enable-dom-distiller |
| 237 - To have a unique user profile every time you run Chrome, you can also add | 272 ``` |
| 238 `--user-data-dir=/tmp/$(mktemp -d)` as a command line parameter. | 273 |
| 239 On Mac OS X, you can instead write | 274 - This adds a menu item `Distill page` that you can use to distill web pages. |
| 240 `--user-data-dir=$(mktemp -d 2>/dev/null || mktemp -d -t 'chromeprofile')`. | 275 |
| 276 - You can also go to `chrome://dom-distiller` to access the debug page. |
| 277 |
| 278 - To have a unique user profile every time you run Chrome, you can also add |
| 279 `--user-data-dir=/tmp/$(mktemp -d)` as a command line parameter. On Mac OS |
| 280 X, you can instead write `--user-data-dir=$(mktemp -d 2>/dev/null || mktemp |
| 281 -d -t 'chromeprofile')`. |
| 241 | 282 |
| 242 ## Running the automated tests in Chromium | 283 ## Running the automated tests in Chromium |
| 243 | 284 |
| 244 - For running the tests, you need to build the `components_browsertests` target: | 285 - For running the tests, you need to build the `components_browsertests` |
| 245 ```bash | 286 target: |
| 246 ninja -C out/Debug components_browsertests | |
| 247 ``` | |
| 248 - Run the `components_browsertests` binary to execute the tests. You can prefix | |
| 249 the command with `xvfb-run` to avoid pop-up windows: | |
| 250 ```bash | |
| 251 xvfb-run out/Debug/components_browsertests | |
| 252 ``` | |
| 253 | 287 |
| 254 - To only run tests related to DOM Distiller, run: | 288 ```bash |
| 255 ```bash | 289 ninja -C out/Debug components_browsertests |
| 256 xvfb-run out/Debug/components_browsertests --gtest_filter=\*Distiller\* | 290 ``` |
| 257 ``` | |
| 258 | 291 |
| 259 - For running tests as isolates, you need to build `components_browsertests_run` | 292 - Run the `components_browsertests` binary to execute the tests. You can |
| 260 and execute them using the swarming tool: | 293 prefix the command with `xvfb-run` to avoid pop-up windows: |
| 261 ```bash | 294 |
| 262 ninja -C out/Debug components_browsertests_run | 295 ```bash |
| 263 python tools/swarming_client/isolate.py run -s out/Debug/components_browsertests
.isolated | 296 xvfb-run out/Debug/components_browsertests |
| 264 ``` | 297 ``` |
| 298 |
| 299 - To only run tests related to DOM Distiller, run: |
| 300 |
| 301 ```bash |
| 302 xvfb-run out/Debug/components_browsertests --gtest_filter=\*Distiller\* |
| 303 ``` |
| 304 |
| 305 - For running tests as isolates, you need to build |
| 306 `components_browsertests_run` and execute them using the swarming tool: |
| 307 |
| 308 ```bash |
| 309 ninja -C out/Debug components_browsertests_run |
| 310 python tools/swarming_client/isolate.py run -s out/Debug/components_browsert
ests.isolated |
| 311 ``` |
| 265 | 312 |
| 266 ## Running the content extractor | 313 ## Running the content extractor |
| 267 | 314 |
| 268 To extract the content from a web page directly, you can run: | 315 To extract the content from a web page directly, you can run: |
| 316 |
| 269 ```bash | 317 ```bash |
| 270 xvfb-run out/Debug/components_browsertests \ | 318 xvfb-run out/Debug/components_browsertests \ |
| 271 --gtest_filter='*MANUAL_ExtractUrl' \ | 319 --gtest_filter='*MANUAL_ExtractUrl' \ |
| 272 --run-manual \ | 320 --run-manual \ |
| 273 --test-tiny-timeout=600000 \ | 321 --test-tiny-timeout=600000 \ |
| 274 --output-file=./extract.out \ | 322 --output-file=./extract.out \ |
| 275 --url=http://www.example.com \ | 323 --url=http://www.example.com \ |
| 276 > ./extract.log 2>&1 | 324 > ./extract.log 2>&1 |
| 277 ``` | 325 ``` |
| 326 |
| 278 `extract.out` has the extracted HTML, `extract.log` has the console logging. | 327 `extract.out` has the extracted HTML, `extract.log` has the console logging. |
| 279 | 328 |
| 280 If you need more logging, you can add the following arguments to the command: | 329 If you need more logging, you can add the following arguments to the command: |
| 281 - Chrome browser: `--vmodule=*distiller*=2` | 330 |
| 282 - Content extractor: `--debug-level=99` | 331 - Chrome browser: `--vmodule=*distiller*=2` |
| 332 - Content extractor: `--debug-level=99` |
| 283 | 333 |
| 284 If this is something you often do, you can put the following function in a bash | 334 If this is something you often do, you can put the following function in a bash |
| 285 file you include (for example `~/.bashrc`) and use it for iterative development: | 335 file you include (for example `~/.bashrc`) and use it for iterative development: |
| 286 | 336 |
| 287 ```bash | 337 ```bash |
| 288 distill() { | 338 distill() { |
| 289 ( | 339 ( |
| 290 roll-distiller && \ | 340 roll-distiller && \ |
| 291 ninja -C out/Debug components_browsertests && | 341 ninja -C out/Debug components_browsertests && |
| 292 xvfb-run out/Debug/components_browsertests \ | 342 xvfb-run out/Debug/components_browsertests \ |
| (...skipping 11 matching lines...) Expand all Loading... |
| 304 | 354 |
| 305 ```bash | 355 ```bash |
| 306 distill http://example.com/article.html | 356 distill http://example.com/article.html |
| 307 ``` | 357 ``` |
| 308 | 358 |
| 309 # Debug Code | 359 # Debug Code |
| 310 | 360 |
| 311 ## Interactive debugging | 361 ## Interactive debugging |
| 312 | 362 |
| 313 You can use the Chrome Developer Tools to debug DOM Distiller: | 363 You can use the Chrome Developer Tools to debug DOM Distiller: |
| 314 - Update the test JavaScript by running `ant gwtc.jstests` or `ant test`. | 364 |
| 315 - Open `war/test.html` in Chrome desktop | 365 - Update the test JavaScript by running `ant gwtc.jstests` or `ant test`. |
| 316 - Open the `Console` panel in Developer Tools (**Ctrl-Shift-J**). | 366 - Open `war/test.html` in Chrome desktop |
| 317 On Mac OS X you can use **⌥-⌘-I** (uppercase `I`) as the shortcut. | 367 - Open the `Console` panel in Developer Tools (**Ctrl-Shift-J**). On Mac OS X |
| 318 - Run all tests by calling: | 368 you can use **⌥-⌘-I** (uppercase `I`) as the shortcut. |
| 319 ```javascript | 369 - Run all tests by calling: |
| 320 org.chromium.distiller.JsTestEntry.run() | 370 |
| 321 ``` | 371 ```javascript |
| 322 - To run only a subset of tests, you can use a regular expression that matches a | 372 org.chromium.distiller.JsTestEntry.run() |
| 323 single test or multiple tests: | 373 ``` |
| 324 ```javascript | 374 |
| 325 org.chromium.distiller.JsTestEntry.runWithFilter('MyTestClass.testSomething') | 375 - To run only a subset of tests, you can use a regular expression that matches |
| 326 ``` | 376 a single test or multiple tests: |
| 377 |
| 378 ```javascript |
| 379 org.chromium.distiller.JsTestEntry.runWithFilter('MyTestClass.testSomething'
) |
| 380 ``` |
| 327 | 381 |
| 328 The `Sources` panel contains both the extracted JavaScript and all the Java | 382 The `Sources` panel contains both the extracted JavaScript and all the Java |
| 329 source files as long as you haven't disabled JavaScript source maps in | 383 source files as long as you haven't disabled JavaScript source maps in Developer |
| 330 Developer Tools. You can set breakpoints in the Java source files and then | 384 Tools. You can set breakpoints in the Java source files and then inspect all |
| 331 inspect all kinds of different interesting things when that breakpoint is hit. | 385 kinds of different interesting things when that breakpoint is hit. |
| 332 | 386 |
| 333 When a test fails, you will see several stack traces. One of these contains | 387 When a test fails, you will see several stack traces. One of these contains |
| 334 clickable links to the corresponding Java source files for the stack frames. | 388 clickable links to the corresponding Java source files for the stack frames. |
| 335 | 389 |
| 336 ## Developer extension | 390 ## Developer extension |
| 337 | 391 |
| 338 After running `ant package`, the `out/extension` folder contains an unpacked | 392 After running `ant package`, the `out/extension` folder contains an unpacked |
| 339 Chrome extension. This can be added to Chrome and used for development. | 393 Chrome extension. This can be added to Chrome and used for development. |
| 340 - Go to `chrome://extensions` | 394 |
| 341 - Enable developer mode | 395 - Go to `chrome://extensions` |
| 342 - Select to load an unpacked extension and point to the `out/extension` folder. | 396 - Enable developer mode |
| 397 - Select to load an unpacked extension and point to the `out/extension` |
| 398 folder. |
| 343 | 399 |
| 344 ### Features | 400 ### Features |
| 345 | 401 |
| 346 The extension currently supports profiling the extraction code. | 402 The extension currently supports profiling the extraction code. |
| 347 | 403 |
| 348 It also adds a panel to the Developer Tools which you can use to trigger | 404 It also adds a panel to the Developer Tools which you can use to trigger |
| 349 extraction on the inspected page. This can be used to trigger and profile | 405 extraction on the inspected page. This can be used to trigger and profile |
| 350 extraction on a mobile device which you are currently inspecting using | 406 extraction on a mobile device which you are currently inspecting using |
| 351 `chrome://inspect`. | 407 `chrome://inspect`. |
| 352 | 408 |
| 353 ## Logging | 409 ## Logging |
| 354 | 410 |
| 355 To add logging, you can use the LogUtil. You can use the Java function | 411 To add logging, you can use the LogUtil. You can use the Java function |
| 356 `LogUtil.logToConsole()`. Destination of logs: | 412 `LogUtil.logToConsole()`. Destination of logs: |
| 357 - `ant test`: Terminal. | 413 |
| 358 To get more verbose output, use `ant test -Dtest.debug_level=99`. | 414 - `ant test`: Terminal. To get more verbose output, use `ant test |
| 359 - Chrome browser: the Chrome log file, as set by shell variable | 415 -Dtest.debug_level=99`. |
| 360 `$CHROME_LOG_FILE`. A release mode build of Chrome will log all JavaScript | 416 - Chrome browser: the Chrome log file, as set by shell variable |
| 361 `INFO` there if you start Chrome with `--enable-logging`. You can add | 417 `$CHROME_LOG_FILE`. A release mode build of Chrome will log all JavaScript |
| 362 `--enable-logging=stderr` to have the log go to stderr instead of a file. | 418 `INFO` there if you start Chrome with `--enable-logging`. You can add |
| 363 - Content extractor: See | 419 `--enable-logging=stderr` to have the log go to stderr instead of a file. |
| 364 [documentation about `extract.log` above](#running-the-content-extractor). | 420 - Content extractor: See [documentation about `extract.log` above] |
| 421 (#running-the-content-extractor). |
| 365 | 422 |
| 366 For an example, see | 423 For an example, see |
| 367 `$DOM_DISTILLER_DIR/java/org/chromium/distiller/PagingLinksFinder.java`. | 424 `$DOM_DISTILLER_DIR/java/org/chromium/distiller/PagingLinksFinder.java`. |
| 368 | 425 |
| 369 Use `ant package '-Dgwt.custom.args=-style PRETTY'` for easier JavaScript | 426 Use `ant package '-Dgwt.custom.args=-style PRETTY'` for easier JavaScript |
| 370 debugging. | 427 debugging. |
| 371 | 428 |
| 372 # Mobile distillation from desktop | 429 # Mobile distillation from desktop |
| 373 | 430 |
| 374 1. In the tab with the interesting URL, bring up the Developer Tools emulation | 431 1. In the tab with the interesting URL, bring up the Developer Tools emulation |
| 375 panel (the mobile device icon). | 432 panel (the mobile device icon). |
| 376 2. Select the desired `Device` and reload the page. Verify that you get what you | 433 2. Select the desired `Device` and reload the page. Verify that you get what |
| 377 expect. For example a Nexus 4 might get a mobile site, whereas Nexus 7 might get | 434 you expect. For example a Nexus 4 might get a mobile site, whereas Nexus 7 |
| 378 the desktop site. | 435 might get the desktop site. |
| 379 3. The User-Agent can be copied directly out from the `UA` field. This field | 436 3. The User-Agent can be copied directly out from the `UA` field. This field |
| 380 does not even require reload after changing device, but it is good practice to | 437 does not even require reload after changing device, but it is good practice |
| 381 verify that you get what you expect. Copy this to the clipboard. | 438 to verify that you get what you expect. Copy this to the clipboard. |
| 382 4. (Re)start chrome with `--user-agent="$USER_AGENT_FROM_CLIPBOARD"`. Remember | 439 4. (Re)start chrome with `--user-agent="$USER_AGENT_FROM_CLIPBOARD"`. Remember |
| 383 to also add `--enable-dom-distiller`. | 440 to also add `--enable-dom-distiller`. |
| 384 5. Distill the same URL in viewer by either using the menu `Distill page` or by | 441 5. Distill the same URL in viewer by either using the menu `Distill page` or by |
| 385 going to `chrome://dom-distiller` and using the input field there. | 442 going to `chrome://dom-distiller` and using the input field there. |
| 386 6. Have fun scrutinizing the Chrome log file. | 443 6. Have fun scrutinizing the Chrome log file. |
| 387 | 444 |
| 388 If you want you can copy some of these User-Agent aliases into normal bash | 445 If you want you can copy some of these User-Agent aliases into normal bash |
| 389 aliases for easy access later. For example, Nexus 4 would be: | 446 aliases for easy access later. For example, Nexus 4 would be: |
| 390 ```bash | 447 |
| 448 ``` |
| 391 --user-agent="Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) Ap
pleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19" | 449 --user-agent="Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) Ap
pleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19" |
| 392 ``` | 450 ``` |
| 393 | 451 |
| 394 Steps 1-3 in the guide above can typically be done in a stable version of | 452 Steps 1-3 in the guide above can typically be done in a stable version of |
| 395 Chrome, whereas the rest of the steps is typically done in your own build of | 453 Chrome, whereas the rest of the steps is typically done in your own build of |
| 396 Chrome (hence the "(Re)" in step 4). Besides speed, this also facilitates | 454 Chrome (hence the "(Re)" in step 4). Besides speed, this also facilitates |
| 397 side-by-side comparison. | 455 side-by-side comparison. |
| OLD | NEW |