Fix final content and title extraction.

Fix final content and title extraction.

The code in ContentExtractor had a slight bug in that it didn't drop
elements that weren't classified as Content. This means that elements
which remained in the dom classified as Boilerplate but with labels
"Maybe Content" etc, would make it into the final output.

Relatedly, we were often merging the <title> and promoting it to "content".
Updated so we don't merge the title with subsequent blocks and ignore
whether it's content

BUG=375449
R=cjhopman@chromium.org

Committed: 88806c2

[Yaron, 2014-05-20 21:02:11]

Could use an eval run on this one to see if it drastically breaks things. I
think it's worth doing but we could hold until I can recover some of the <title>
breakage.

[Yaron, 2014-05-21 22:29:06]

On 2014/05/20 21:02:11, Yaron wrote:
> Could use an eval run on this one to see if it drastically breaks things. I
> think it's worth doing but we could hold until I can recover some of the
<title>
> breakage.

This accidentally landed as part of https://codereview.chromium.org/291823005/.

After playing around with it, I realized that it's definitely causing the
<title> to stick around too often as the <title> often gets merged with
"CONTENT" and then isn't stripped. I think we should ensure that TITLE doesn't
merge with other nodes.

[Yaron, 2014-05-22 00:22:35]

trying again, this time a test. Local testing seems to do better

[cjhopman, 2014-05-22 00:27:21]

https://codereview.chromium.org/290993004/diff/20001/src/com/dom_distiller/cl...
File src/com/dom_distiller/client/ContentExtractor.java (right):

https://codereview.chromium.org/290993004/diff/20001/src/com/dom_distiller/cl...
src/com/dom_distiller/client/ContentExtractor.java:76:
LogUtil.logToConsole(clonedSubtree.getNodeType() + " node:" +
Node.ELEMENT_NODE);
probably don't need this. If it's been useful, at least guard it with some debug
flag.

https://codereview.chromium.org/290993004/diff/20001/test/com/dom_distiller/c...
File test/com/dom_distiller/client/ContentExtractorTest.java (right):

https://codereview.chromium.org/290993004/diff/20001/test/com/dom_distiller/c...
test/com/dom_distiller/client/ContentExtractorTest.java:44: String
extractedContent = ContentExtractor.extractContent();
Could we confirm that this isn't extracted because it's the title. Something
like:

Setup (without Document.get.setTitle())
extractContent()
check extracted content
Document.get().setTitle(TITLE_TEXT)
extractContent()
check extracted content

[Yaron, 2014-05-22 17:05:24]

https://codereview.chromium.org/290993004/diff/20001/src/com/dom_distiller/cl...
File src/com/dom_distiller/client/ContentExtractor.java (right):

https://codereview.chromium.org/290993004/diff/20001/src/com/dom_distiller/cl...
src/com/dom_distiller/client/ContentExtractor.java:76:
LogUtil.logToConsole(clonedSubtree.getNodeType() + " node:" +
Node.ELEMENT_NODE);
On 2014/05/22 00:27:21, cjhopman wrote:
> probably don't need this. If it's been useful, at least guard it with some
debug
> flag.

err right that was supposed to go. I was trying to investigate something with
small pages. I believe you can fail the below test if we return just a single
text node but can't repro it now.

https://codereview.chromium.org/290993004/diff/20001/test/com/dom_distiller/c...
File test/com/dom_distiller/client/ContentExtractorTest.java (right):

https://codereview.chromium.org/290993004/diff/20001/test/com/dom_distiller/c...
test/com/dom_distiller/client/ContentExtractorTest.java:44: String
extractedContent = ContentExtractor.extractContent();
On 2014/05/22 00:27:21, cjhopman wrote:
> Could we confirm that this isn't extracted because it's the title. Something
> like:
> 
> Setup (without Document.get.setTitle())
> extractContent()
> check extracted content
> Document.get().setTitle(TITLE_TEXT)
> extractContent()
> check extracted content

Done.

[cjhopman, 2014-05-22 19:01:36]

lgtm

[kuan, 2014-05-22 19:13:34]

fyi: i patched this in, and tried to distill 2 of the urls i was using for
tables.

- http://donnafisherjackson.com/services/enneagram_relationships.html.  the bold
text at the top "Introduction - an article series..."  and the table heading
"Type" are gone.  without these text, the table is not considered relevant, and
won't be extracted if we want to extract tables in content.

- http://jackiesguineapiggies.com/bathingyourguineapig.html.  the bold text at
the top "Bathing your guinea pig" is gone.  so the table that follows also won't
be extracted.  which ironically, is what we really want, since the table is used
for layout.

[Yaron, 2014-05-22 20:14:38]

Committed patchset #3 manually as r88806c2 (presubmit successful).