Add LOAD_PRERENDER to prerender motivated load requests

Add LOAD_PRERENDER to prerender motivated load requests.

This CL depends on this webkit CL landing first: https://bugs.webkit.org/show_bug.cgi?id=54744

And also it's built based on http://codereview.chromium.org/6532031/ , and so that CL should land first too.

BUG=73418
TEST=not yet

[gavinp, 2011-02-18 15:00:26]

Here it is: it turns out to be not that scary after all.

[mmenke, 2011-02-18 15:12:07]

LGTM (After countless hours spent reviewing this massive patch.  I think perhaps
it should have been broken down into 3 or 4 separate patches)

http://codereview.chromium.org/6542016/diff/5001/net/base/load_flags_list.h
File net/base/load_flags_list.h (right):

http://codereview.chromium.org/6542016/diff/5001/net/base/load_flags_list.h#n...
net/base/load_flags_list.h:106: // and is not intended for display.
nit:  Might want to change that to "immediate display", since the idea is to
prerender things that are likely to be displayed, as opposed to random things we
have no intention of ever displaying.

[cbentzel, 2011-02-18 15:50:14]

LGTM

Please get brett or Darin to review, after dependencies are through, however.

Would be nice to add a COMPILE_ASSERT on max LOAD_FLAG number while your at it -
although an enum, the ViewHostMsg_Resource_Request serializes as an int, and
we're starting to get close to 32 values in the bitmap now.

[dominich, 2011-02-18 16:04:06]

LGTM

On 2011/02/18 15:00:26, gavinp wrote:
> Here it is: it turns out to be not that scary after all.

[gavinp, 2011-02-18 19:06:43]

Fixed the comment.  I'll invite brettw or Darin by after the WebKit change
lands.

[darin (slow to review), 2011-02-18 22:24:06]

See my comments here:
https://bugs.webkit.org/show_bug.cgi?id=54744#c10

I don't think this the WebKit plumbing is necessary.  You can deduce what you
need from the routing ID.

[darin (slow to review), 2011-02-18 22:24:16]

On 2011/02/18 22:24:06, darin wrote:
> See my comments here:
> https://bugs.webkit.org/show_bug.cgi?id=54744#c10
> 
> I don't think this the WebKit plumbing is necessary.  You can deduce what you
> need from the routing ID.

Sorry for not thinking of this sooner!!

[cbentzel, 2011-02-18 22:34:07]

We discussed doing this in the browser side based on route id. There was
some complexity with maintaining the map in the io thread (since the
prerendercontents are created and destroyed in ui thread) but it may be the
easier option.

Could we do this at ipcresourceloaderbridge in the render process, after the
request has gone through webkit but before sent to browser?

It would be nice to minimize the number of places that have to maintain
prerendered state.
On Feb 18, 2011 5:24 PM, <darin@chromium.org> wrote:
> On 2011/02/18 22:24:06, darin wrote:
>> See my comments here:
>> https://bugs.webkit.org/show_bug.cgi?id=54744#c10
>
>> I don't think this the WebKit plumbing is necessary. You can deduce what
>> you
>> need from the routing ID.
>
> Sorry for not thinking of this sooner!!
>
>
> http://codereview.chromium.org/6542016/

[darin (slow to review), 2011-02-18 23:32:30]

Yeah, I don't know where the best place for the mapping to live is.  I know
that the document request for the pre-render always comes first, so you
could assign to the prerender set directly on the IO thread.  You would just
need to know when to clear the entries, but that could happen when the
RenderViewHost is closed.  It could post a task to the IO thread to clear
the set.  No need for locking!

-Darin


On Fri, Feb 18, 2011 at 2:33 PM, Chris Bentzel <cbentzel@chromium.org>wrote:

> We discussed doing this in the browser side based on route id. There was
> some complexity with maintaining the map in the io thread (since the
> prerendercontents are created and destroyed in ui thread) but it may be the
> easier option.
>
> Could we do this at ipcresourceloaderbridge in the render process, after
> the request has gone through webkit but before sent to browser?
>
> It would be nice to minimize the number of places that have to maintain
> prerendered state.
> On Feb 18, 2011 5:24 PM, <darin@chromium.org> wrote:
> > On 2011/02/18 22:24:06, darin wrote:
> >> See my comments here:
> >> https://bugs.webkit.org/show_bug.cgi?id=54744#c10
> >
> >> I don't think this the WebKit plumbing is necessary. You can deduce what
>
> >> you
> >> need from the routing ID.
> >
> > Sorry for not thinking of this sooner!!
> >
> >
> > http://codereview.chromium.org/6542016/
>

[gavinp, 2011-02-19 03:31:58]

On 2011/02/18 23:32:30, darin wrote:
> Yeah, I don't know where the best place for the mapping to live is.  I know
> that the document request for the pre-render always comes first, so you
> could assign to the prerender set directly on the IO thread.  You would just
> need to know when to clear the entries, but that could happen when the
> RenderViewHost is closed.  It could post a task to the IO thread to clear
> the set.  No need for locking!

We actually had two competing implementations written up for solving this
problem!  Dominic had written up some code to do the mapping & tagging in the
ResourceDispatcherHost; although he was using a system of posting between the UI
to the IO thread and back again; which avoided locking and races.  I'm
interested in using the document request for the pre-render to trigger...  But I
don't fully get how that would work.  We submitted this CL and not the other one
because we didn't like the posting.  Probably it should at least be uploaded now
to look at...

Over in the webkit bug 54744, this issue did revive discussion of webkit bug
49113, created by Darin Fisher, which discusses creating a mechanism for moving
arbitrary data around with a resource request.  I think if we solved that
puzzle, we'd probably be more comfortable adding information like this to travel
through with requests.

There's two other use cases that have motivated me to want a way to connect
subresource requests with their originating main resource request.  One is
favicon auth, which cbentzel can talk about better than me.  The other is tab
associating requests.  If we make the network prioritization more sophisticated,
we'd likely consider lowering the priority of invisible tabs when visible tabs
are making requests.  As I consider that use, I find myself liking the RDH as a
home for this information: it's more likely to be able to for instance upgrade
pending requests (which after all might be long lived XMLHttpRequests etc...)
when tabs become visible.


> 
> -Darin
> 
> 
> On Fri, Feb 18, 2011 at 2:33 PM, Chris Bentzel <cbentzel@chromium.org>wrote:
> 
> > We discussed doing this in the browser side based on route id. There was
> > some complexity with maintaining the map in the io thread (since the
> > prerendercontents are created and destroyed in ui thread) but it may be the
> > easier option.
> >
> > Could we do this at ipcresourceloaderbridge in the render process, after
> > the request has gone through webkit but before sent to browser?
> >
> > It would be nice to minimize the number of places that have to maintain
> > prerendered state.
> > On Feb 18, 2011 5:24 PM, <mailto:darin@chromium.org> wrote:
> > > On 2011/02/18 22:24:06, darin wrote:
> > >> See my comments here:
> > >> https://bugs.webkit.org/show_bug.cgi?id=54744#c10
> > >
> > >> I don't think this the WebKit plumbing is necessary. You can deduce what
> >
> > >> you
> > >> need from the routing ID.
> > >
> > > Sorry for not thinking of this sooner!!
> > >
> > >
> > > http://codereview.chromium.org/6542016/
> >

[darin (slow to review), 2011-02-19 05:13:00]

On Fri, Feb 18, 2011 at 7:31 PM, <gavinp@chromium.org> wrote:

> On 2011/02/18 23:32:30, darin wrote:
>
>> Yeah, I don't know where the best place for the mapping to live is.  I
>> know
>> that the document request for the pre-render always comes first, so you
>> could assign to the prerender set directly on the IO thread.  You would
>> just
>> need to know when to clear the entries, but that could happen when the
>> RenderViewHost is closed.  It could post a task to the IO thread to clear
>> the set.  No need for locking!
>>
>
> We actually had two competing implementations written up for solving this
> problem!  Dominic had written up some code to do the mapping & tagging in
> the
> ResourceDispatcherHost; although he was using a system of posting between
> the UI
> to the IO thread and back again; which avoided locking and races.  I'm
> interested in using the document request for the pre-render to trigger...
>  But I
> don't fully get how that would work.  We submitted this CL and not the
> other one
> because we didn't like the posting.  Probably it should at least be
> uploaded now
> to look at...
>
> Over in the webkit bug 54744, this issue did revive discussion of webkit
> bug
> 49113, created by Darin Fisher, which discusses creating a mechanism for
> moving
> arbitrary data around with a resource request.  I think if we solved that
> puzzle, we'd probably be more comfortable adding information like this to
> travel
> through with requests.
>
> There's two other use cases that have motivated me to want a way to connect
> subresource requests with their originating main resource request.  One is
> favicon auth, which cbentzel can talk about better than me.  The other is
> tab
> associating requests.  If we make the network prioritization more
> sophisticated,
> we'd likely consider lowering the priority of invisible tabs when visible
> tabs
> are making requests.  As I consider that use, I find myself liking the RDH
> as a
> home for this information: it's more likely to be able to for instance
> upgrade
> pending requests (which after all might be long lived XMLHttpRequests
> etc...)
> when tabs become visible.
>
>
All of these seem like they are well solved by knowing the routing ID.  The
routing ID
identifies the tab.

Plumbing extra request parameters seems redundant.  All we need is a
single mapping
of view IDs to properties.  Maybe we can generalize this.

-Darin




>
>
>
>  -Darin
>>
>
>
>  On Fri, Feb 18, 2011 at 2:33 PM, Chris Bentzel <cbentzel@chromium.org
>> >wrote:
>>
>
>  > We discussed doing this in the browser side based on route id. There was
>> > some complexity with maintaining the map in the io thread (since the
>> > prerendercontents are created and destroyed in ui thread) but it may be
>> the
>> > easier option.
>> >
>> > Could we do this at ipcresourceloaderbridge in the render process, after
>> > the request has gone through webkit but before sent to browser?
>> >
>> > It would be nice to minimize the number of places that have to maintain
>> > prerendered state.
>> > On Feb 18, 2011 5:24 PM, <mailto:darin@chromium.org> wrote:
>> > > On 2011/02/18 22:24:06, darin wrote:
>> > >> See my comments here:
>> > >> https://bugs.webkit.org/show_bug.cgi?id=54744#c10
>> > >
>> > >> I don't think this the WebKit plumbing is necessary. You can deduce
>> what
>> >
>> > >> you
>> > >> need from the routing ID.
>> > >
>> > > Sorry for not thinking of this sooner!!
>> > >
>> > >
>> > > http://codereview.chromium.org/6542016/
>> >
>>
>
>
>
> http://codereview.chromium.org/6542016/
>

[gavinp, 2011-02-19 11:12:34]

On 2011/02/19 05:13:00, darin wrote:
> On Fri, Feb 18, 2011 at 7:31 PM, <mailto:gavinp@chromium.org> wrote:
> 
> > On 2011/02/18 23:32:30, darin wrote:
> >
> >> Yeah, I don't know where the best place for the mapping to live is.  I
> >> know
> >> that the document request for the pre-render always comes first, so you
> >> could assign to the prerender set directly on the IO thread.  You would
> >> just
> >> need to know when to clear the entries, but that could happen when the
> >> RenderViewHost is closed.  It could post a task to the IO thread to clear
> >> the set.  No need for locking!
> >>
> >
> > We actually had two competing implementations written up for solving this
> > problem!  Dominic had written up some code to do the mapping & tagging in
> > the
> > ResourceDispatcherHost; although he was using a system of posting between
> > the UI
> > to the IO thread and back again; which avoided locking and races.  I'm
> > interested in using the document request for the pre-render to trigger...
> >  But I
> > don't fully get how that would work.  We submitted this CL and not the
> > other one
> > because we didn't like the posting.  Probably it should at least be
> > uploaded now
> > to look at...
> >
> > Over in the webkit bug 54744, this issue did revive discussion of webkit
> > bug
> > 49113, created by Darin Fisher, which discusses creating a mechanism for
> > moving
> > arbitrary data around with a resource request.  I think if we solved that
> > puzzle, we'd probably be more comfortable adding information like this to
> > travel
> > through with requests.
> >
> > There's two other use cases that have motivated me to want a way to connect
> > subresource requests with their originating main resource request.  One is
> > favicon auth, which cbentzel can talk about better than me.  The other is
> > tab
> > associating requests.  If we make the network prioritization more
> > sophisticated,
> > we'd likely consider lowering the priority of invisible tabs when visible
> > tabs
> > are making requests.  As I consider that use, I find myself liking the RDH
> > as a
> > home for this information: it's more likely to be able to for instance
> > upgrade
> > pending requests (which after all might be long lived XMLHttpRequests
> > etc...)
> > when tabs become visible.
> >
> >
> All of these seem like they are well solved by knowing the routing ID.  The
> routing ID
> identifies the tab.
> 
> Plumbing extra request parameters seems redundant.  All we need is a
> single mapping
> of view IDs to properties.  Maybe we can generalize this.
> 

General, and in the browser process seems like it'd be most useful for my
purposes.  I'm going to think about this, maybe get something to upload.

- Gavin

> -Darin
> 
> 
> 
> 
> >
> >
> >
> >  -Darin
> >>
> >
> >
> >  On Fri, Feb 18, 2011 at 2:33 PM, Chris Bentzel
<mailto:cbentzel@chromium.org
> >> >wrote:
> >>
> >
> >  > We discussed doing this in the browser side based on route id. There was
> >> > some complexity with maintaining the map in the io thread (since the
> >> > prerendercontents are created and destroyed in ui thread) but it may be
> >> the
> >> > easier option.
> >> >
> >> > Could we do this at ipcresourceloaderbridge in the render process, after
> >> > the request has gone through webkit but before sent to browser?
> >> >
> >> > It would be nice to minimize the number of places that have to maintain
> >> > prerendered state.
> >> > On Feb 18, 2011 5:24 PM, <mailto:darin@chromium.org> wrote:
> >> > > On 2011/02/18 22:24:06, darin wrote:
> >> > >> See my comments here:
> >> > >> https://bugs.webkit.org/show_bug.cgi?id=54744#c10
> >> > >
> >> > >> I don't think this the WebKit plumbing is necessary. You can deduce
> >> what
> >> >
> >> > >> you
> >> > >> need from the routing ID.
> >> > >
> >> > > Sorry for not thinking of this sooner!!
> >> > >
> >> > >
> >> > > http://codereview.chromium.org/6542016/
> >> >
> >>
> >
> >
> >
> > http://codereview.chromium.org/6542016/
> >

[cbentzel, 2011-02-19 14:34:35]

Also - I think we could use this mapping eventually to lower resource
priorities for non-foreground tabs [except perhaps audio resources and a few
other cases].

On Sat, Feb 19, 2011 at 6:12 AM, <gavinp@chromium.org> wrote:

> On 2011/02/19 05:13:00, darin wrote:
>
>  On Fri, Feb 18, 2011 at 7:31 PM, <mailto:gavinp@chromium.org> wrote:
>>
>
>  > On 2011/02/18 23:32:30, darin wrote:
>> >
>> >> Yeah, I don't know where the best place for the mapping to live is.  I
>> >> know
>> >> that the document request for the pre-render always comes first, so you
>> >> could assign to the prerender set directly on the IO thread.  You would
>> >> just
>> >> need to know when to clear the entries, but that could happen when the
>> >> RenderViewHost is closed.  It could post a task to the IO thread to
>> clear
>> >> the set.  No need for locking!
>> >>
>> >
>> > We actually had two competing implementations written up for solving
>> this
>> > problem!  Dominic had written up some code to do the mapping & tagging
>> in
>> > the
>> > ResourceDispatcherHost; although he was using a system of posting
>> between
>> > the UI
>> > to the IO thread and back again; which avoided locking and races.  I'm
>> > interested in using the document request for the pre-render to
>> trigger...
>> >  But I
>> > don't fully get how that would work.  We submitted this CL and not the
>> > other one
>> > because we didn't like the posting.  Probably it should at least be
>> > uploaded now
>> > to look at...
>> >
>> > Over in the webkit bug 54744, this issue did revive discussion of webkit
>> > bug
>> > 49113, created by Darin Fisher, which discusses creating a mechanism for
>> > moving
>> > arbitrary data around with a resource request.  I think if we solved
>> that
>> > puzzle, we'd probably be more comfortable adding information like this
>> to
>> > travel
>> > through with requests.
>> >
>> > There's two other use cases that have motivated me to want a way to
>> connect
>> > subresource requests with their originating main resource request.  One
>> is
>> > favicon auth, which cbentzel can talk about better than me.  The other
>> is
>> > tab
>> > associating requests.  If we make the network prioritization more
>> > sophisticated,
>> > we'd likely consider lowering the priority of invisible tabs when
>> visible
>> > tabs
>> > are making requests.  As I consider that use, I find myself liking the
>> RDH
>> > as a
>> > home for this information: it's more likely to be able to for instance
>> > upgrade
>> > pending requests (which after all might be long lived XMLHttpRequests
>> > etc...)
>> > when tabs become visible.
>> >
>> >
>> All of these seem like they are well solved by knowing the routing ID.
>>  The
>> routing ID
>> identifies the tab.
>>
>
>  Plumbing extra request parameters seems redundant.  All we need is a
>> single mapping
>> of view IDs to properties.  Maybe we can generalize this.
>>
>
>
> General, and in the browser process seems like it'd be most useful for my
> purposes.  I'm going to think about this, maybe get something to upload.
>
> - Gavin
>
>
>  -Darin
>>
>
>
>
>
>  >
>> >
>> >
>> >  -Darin
>> >>
>> >
>> >
>> >  On Fri, Feb 18, 2011 at 2:33 PM, Chris Bentzel
>>
> <mailto:cbentzel@chromium.org
>
>  >> >wrote:
>> >>
>> >
>> >  > We discussed doing this in the browser side based on route id. There
>> was
>> >> > some complexity with maintaining the map in the io thread (since the
>> >> > prerendercontents are created and destroyed in ui thread) but it may
>> be
>> >> the
>> >> > easier option.
>> >> >
>> >> > Could we do this at ipcresourceloaderbridge in the render process,
>> after
>> >> > the request has gone through webkit but before sent to browser?
>> >> >
>> >> > It would be nice to minimize the number of places that have to
>> maintain
>> >> > prerendered state.
>> >> > On Feb 18, 2011 5:24 PM, <mailto:darin@chromium.org> wrote:
>> >> > > On 2011/02/18 22:24:06, darin wrote:
>> >> > >> See my comments here:
>> >> > >> https://bugs.webkit.org/show_bug.cgi?id=54744#c10
>> >> > >
>> >> > >> I don't think this the WebKit plumbing is necessary. You can
>> deduce
>> >> what
>> >> >
>> >> > >> you
>> >> > >> need from the routing ID.
>> >> > >
>> >> > > Sorry for not thinking of this sooner!!
>> >> > >
>> >> > >
>> >> > > http://codereview.chromium.org/6542016/
>> >> >
>> >>
>> >
>> >
>> >
>> > http://codereview.chromium.org/6542016/
>> >
>>
>
>
>
> http://codereview.chromium.org/6542016/
>

[cbentzel, 2011-03-03 15:20:20]

I think it's safe to close this issue now. The browser-process only version from
dominich got an LGTM from myself and brettw.