Renderer-side changes for NoState Prefetch

third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp

Index: third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp
diff --git a/third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp b/third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp
index 07f747367118d5231744ac8252c9c26bf526b0f6..d90eef3c8038302c570595001bde62b5eaa7246b 100644
--- a/third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp
+++ b/third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp
@@ -89,6 +89,17 @@ static HTMLTokenizer::State tokenizerStateForContextElement(Element* contextElem
     return HTMLTokenizer::DataState;
 }
 
+// static
+HTMLDocumentParser* HTMLDocumentParser::create(

[Charlie Harrison, 2016/08/03 14:40:03]

I have a slight preference for passing the sync policy from the callsite,
instead of reaching into the document in threadingAllowedForDocument.

[droger, 2016/08/03 16:45:50]

Done.

+    HTMLDocument& document,
+    ParserSynchronizationPolicy backgroundParsingPolicy)
+{
+    return new HTMLDocumentParser(
+        document,
+        threadingAllowedForDocument(document)
+            ? backgroundParsingPolicy : ForceSynchronousParsing);
+}
+
 HTMLDocumentParser::HTMLDocumentParser(HTMLDocument& document, ParserSynchronizationPolicy syncPolicy)
     : HTMLDocumentParser(document, AllowScriptingContent, syncPolicy)
 {
@@ -150,6 +161,12 @@ DEFINE_TRACE(HTMLDocumentParser)
     HTMLScriptRunnerHost::trace(visitor);
 }
 
+// static
+bool HTMLDocumentParser::threadingAllowedForDocument(HTMLDocument& document)
+{
+    return !document.isPrefetchOnly();
+}
+
 void HTMLDocumentParser::detach()
 {
     if (!isParsingFragment() && m_tokenizedChunkQueue.get() && m_tokenizedChunkQueue->peakPendingChunkCount()) {
@@ -707,7 +724,8 @@ void HTMLDocumentParser::insert(const SegmentedString& source)
         if (!m_insertionPreloadScanner)
             m_insertionPreloadScanner = createPreloadScanner();
         m_insertionPreloadScanner->appendToEnd(source);
-        m_insertionPreloadScanner->scanAndPreload(m_preloader.get(), document()->validBaseElementURL(), nullptr);
+        m_insertionPreloadScanner->scanAndPreload(

[Charlie Harrison, 2016/08/03 14:40:03]

nit: Unnecessary line break.

[droger, 2016/08/03 16:45:50]

Done.

+            m_preloader.get(), document()->validBaseElementURL(), nullptr);
     }
 
     endIfDelayed();
@@ -788,6 +806,17 @@ void HTMLDocumentParser::append(const String& inputSource)
     TRACE_EVENT1(TRACE_DISABLED_BY_DEFAULT("blink.debug"), "HTMLDocumentParser::append", "size", inputSource.length());
     const SegmentedString source(inputSource);
 
+    if (document()->isPrefetchOnly()) {
+        if (!m_preloadScanner)
+            m_preloadScanner = createPreloadScanner();
+
+        m_preloadScanner->appendToEnd(source);
+        m_preloadScanner->scanAndPreload(m_preloader.get(), document()->validBaseElementURL(), nullptr);

[Charlie Harrison, 2016/08/03 14:40:03]

This just preloads the tokens normally. Do we want to add load flags like
LOAD_PREFETCH so all resources are forced into the HTTP cache?

Also, if we aren't using this render process for the final page load, this will
bloat the MemoryCache with in-memory resources that will never be used.

+shivanisha@ is thinking of implementing a "fire-and-forget" system in Blink
that will send requests but ignore results (make sure they don't land in the
memory cache). It might make sense to collaborate on this.

(note: I understand this is just a prototype, I just want to understand what the
final design will look like.)

[droger, 2016/08/03 16:45:50]

This uses the same code as the standard preload scanner. I would think that it
uses the LOAD_PREFETCH flags accordingly, and (at first at least) we want to
have the same behavior for prefetch.
Yes, this is a good point. Disabling the memory cache would probably be nice.

[Charlie Harrison, 2016/08/03 17:15:14]

Nope, LOAD_PREFETCH afaik is only used for the link rel="prefetch" feature.
These loads will look like normal requests from the net stack, and will not
follow special caching behavior like LOAD_PREFETCH.

I did a brief trace and you can see that the load flag is added when the request
has RequestContext::RequestContextPrefetch which only happens for link rel
prefetches. One thing that you could do in this patch to solve this problem is
to inspect the preloads that are being scanned, and set their type to
LinkRelPrefetch. This would require:
1. Adding "setForPrefetch" to PreloadRequest.h which sets the resource type to
LinkRelPrefetch.
2. one of:
 a. Modifying HTMLPreloadScanner in some way to do this for you.
 b. Using a TokenPreloadScanner instead of HTMLPreloadScanner (look at
HTMLPreloadScanner for how to do this)

If you'd like LOAD_PREFETCH on your requests, you'll need to do this otherwise
you will double download no-store resources, etc.

This technique sounds reasonable to me, but again, I am not an owner so you'll
need to get sign-off from some blink owner :)

[pasko, 2016/08/04 12:23:08]

csharrison: thank you for investigation and the info. Let's handle load flags
properly in a separate patch. I indeed see that LOAD_PREFETCH is not set in the
net stack, but strangely the following page load still fetches from cache
ignoring validators. Hmm.. I thought only LOAD_PREFETCH can do that..

+
+        // Return after the preload scanner, do not actually parse the document.
+        return;
+    }
+
     if (m_preloadScanner) {
         if (m_input.current().isEmpty() && !isWaitingForScripts()) {
             // We have parsed until the end of the current input and so are now moving ahead of the preload scanner.