Initial child-process side plumbing for starting an embedded ServiceWorker context

Initial child-process side plumbing for starting an embedded ServiceWorker context

Design assumptions (some assumptions can be tentative) for this CL:
* ServiceWorkerProvider is going to be embedded in all frame/document
* Browser process can tell which SWP is associated to which SW (if any),
  via SWP registration process that is not added yet
* A newly created embedded worker context will always create its own shadow page
 for its loading context
* A newly created embedded worker context stub will keep a ref to the embedding process

No browser-side implementation / blink plumbing yet.

This adds a new tiny dispatcher which listens messages
solely on the renderer thread.

BUG=313530
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=237873

[kinuko, 2013-10-31 12:16:24]

Hi, before writing too much code I'd like to start with a smaller child-process
only patch, to we can share/agree on
the same design assumptions. PTL

[michaeln, 2013-10-31 23:31:36]

I'm looking at the CL description and starting there.

Design assumptions (some assumptions can be tentative) for this CL: 
* ServiceWorkerProvider is going to be embedded in all frame/document

Yup, also each SharedWorker/EmbeddedWorker should have a distinct provider.

* Browser process can tell which SWP is associated to which SW (if any), 
via SWP registration process that is not added yet 

Yup.

* We embed/collocate a new embedded worker context only in renderer processes

Yup.

* A newly created embedded worker context will use the SWP's document context 
as its loader context (used in WorkerLoaderProxy) 

No. I don't think that works well for a variety of reasons.
- requests made by document subject to interception by the serviceworker
instance, request made by the serviceworker itself are not, so they shouldn't be
using the same loader.
- when navigating away from the document, we don't want to have to shutdown the
serviceworker instance or terminate any of its pending request
- requests made by serviceworker should not show up in the inspector for the
page

I think we have to arrange for each service worker to have its own distinct
loading context separate from any document. For SharedWorkers the shadow page /
document provides that distinct context. See
WebSharedWorkerImpl::initializeLoader.

* A newly created embedded worker context stub will keep a ref to the embedding 
renderer process (as well as loader context's ExecutionContext)

Yup process ref.
Nope ref to other execontext of a particular controlled document.

[kinuko, 2013-10-31 23:56:26]

On 2013/10/31 23:31:36, michaeln wrote:
> I'm looking at the CL description and starting there.
> 
> Design assumptions (some assumptions can be tentative) for this CL: 
> * ServiceWorkerProvider is going to be embedded in all frame/document
> 
> Yup, also each SharedWorker/EmbeddedWorker should have a distinct provider.
> 
> * Browser process can tell which SWP is associated to which SW (if any), 
> via SWP registration process that is not added yet 
> 
> Yup.
> 
> * We embed/collocate a new embedded worker context only in renderer processes
> 
> Yup.
> 
> * A newly created embedded worker context will use the SWP's document context 
> as its loader context (used in WorkerLoaderProxy) 
> 
> No. I don't think that works well for a variety of reasons.
> - requests made by document subject to interception by the serviceworker
> instance, request made by the serviceworker itself are not, so they shouldn't
be
> using the same loader.
> - when navigating away from the document, we don't want to have to shutdown
the
> serviceworker instance or terminate any of its pending request
> - requests made by serviceworker should not show up in the inspector for the
> page
> 
> I think we have to arrange for each service worker to have its own distinct
> loading context separate from any document. For SharedWorkers the shadow page
/
> document provides that distinct context. See
> WebSharedWorkerImpl::initializeLoader.

Ah, thanks this was the biggest unclear point to me I wanted to make sure
earlier.
I wanted to remove shadow page if possible but also wandered it might be cleaner
to create a new shadow context for each SW context, but just wasn't sure if it's
necessary.  The fact that loading in the SWP context was intercepted by SW was a
very good point that I had overlooked.

> * A newly created embedded worker context stub will keep a ref to the
embedding 
> renderer process (as well as loader context's ExecutionContext)
> 
> Yup process ref.
> Nope ref to other execontext of a particular controlled document.

Ok, thanks!  I'll revise this CL.

[kinuko, 2013-11-01 13:20:12]

I updated the design assumptions in the issue description and updated the CL. 
As you can see this CL contains very minimum code (mainly for placeholders) for
now.  PTAL thanks!

[michaeln, 2013-11-04 22:04:51]

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
File content/child/service_worker/embedded_worker_dispatcher.cc (right):

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
content/child/service_worker/embedded_worker_dispatcher.cc:29: new
EmbeddedWorkerStub(script_url, route_id);
Its odd to see something newed and the ptr value not stored somewhere.

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
File content/child/service_worker/embedded_worker_dispatcher.h (right):

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
content/child/service_worker/embedded_worker_dispatcher.h:25: void
OnCreateWorker(const GURL& script_url, int route_id);
I think for serviceworkers we'll need other params besides original url. We'll
need to identify the right 'instance' somehow.

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
File content/child/service_worker/embedded_worker_stub.cc (right):

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
content/child/service_worker/embedded_worker_stub.cc:17:
ChildThread::current()->AddRoute(route_id, this);
The 'routed message' infrastructure is great for delivering messages to the
OnMessageReceived method on the main thread, are we going to be using that for
these guys?

I see this as an analog to websharedworker_stub.cc.

In most cases, I think we can avoid switching to the 'main' thread only to
switch threads again to a background thread and should strive to do so. Two
things are unavoidably involve the main thread...
- starting a new worker (handled by your dispatcher class in this CL)
- terminiating an existing worker (could also be handled by the dispatcher
class)

I guess what i'm getting at is that maybe we don't need to use the rounted
message infrastructure for this stuff. (Although, I see the devtools integration
done websharedworker_stub.cc might depend on it in some way?).

[kinuko, 2013-11-05 01:01:51]

Some quick responses

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
File content/child/service_worker/embedded_worker_dispatcher.h (right):

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
content/child/service_worker/embedded_worker_dispatcher.h:25: void
OnCreateWorker(const GURL& script_url, int route_id);
On 2013/11/04 22:04:51, michaeln wrote:
> I think for serviceworkers we'll need other params besides original url. We'll
> need to identify the right 'instance' somehow.

Sure, I had several more parameters for this but deleted them for now, as I
wasn't sure what value would be exactly needed. (I should have left some TODO
comments here)

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
File content/child/service_worker/embedded_worker_stub.cc (right):

https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
content/child/service_worker/embedded_worker_stub.cc:17:
ChildThread::current()->AddRoute(route_id, this);
On 2013/11/04 22:04:51, michaeln wrote:
> The 'routed message' infrastructure is great for delivering messages to the
> OnMessageReceived method on the main thread, are we going to be using that for
> these guys?
> 
> I see this as an analog to websharedworker_stub.cc.
> 
> In most cases, I think we can avoid switching to the 'main' thread only to
> switch threads again to a background thread and should strive to do so. Two
> things are unavoidably involve the main thread...
> - starting a new worker (handled by your dispatcher class in this CL)
> - terminiating an existing worker (could also be handled by the dispatcher
> class)
> 
> I guess what i'm getting at is that maybe we don't need to use the rounted
> message infrastructure for this stuff. (Although, I see the devtools
integration
> done websharedworker_stub.cc might depend on it in some way?).

I expected having each worker's own routing might make code simpler, but I'm
sure we can likely do the same without routing too.  What concerns you most is
threading overhead btw?  I can pursue the model (no routing model) too.

[kinuko, 2013-11-05 08:24:26]

On 2013/11/05 01:01:51, kinuko wrote:
> Some quick responses
> 
>
https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
> File content/child/service_worker/embedded_worker_dispatcher.h (right):
> 
>
https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
> content/child/service_worker/embedded_worker_dispatcher.h:25: void
> OnCreateWorker(const GURL& script_url, int route_id);
> On 2013/11/04 22:04:51, michaeln wrote:
> > I think for serviceworkers we'll need other params besides original url.
We'll
> > need to identify the right 'instance' somehow.
> 
> Sure, I had several more parameters for this but deleted them for now, as I
> wasn't sure what value would be exactly needed. (I should have left some TODO
> comments here)
> 
>
https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
> File content/child/service_worker/embedded_worker_stub.cc (right):
> 
>
https://codereview.chromium.org/54573002/diff/540001/content/child/service_wo...
> content/child/service_worker/embedded_worker_stub.cc:17:
> ChildThread::current()->AddRoute(route_id, this);
> On 2013/11/04 22:04:51, michaeln wrote:
> > The 'routed message' infrastructure is great for delivering messages to the
> > OnMessageReceived method on the main thread, are we going to be using that
for
> > these guys?
> > 
> > I see this as an analog to websharedworker_stub.cc.
> > 
> > In most cases, I think we can avoid switching to the 'main' thread only to
> > switch threads again to a background thread and should strive to do so. Two
> > things are unavoidably involve the main thread...
> > - starting a new worker (handled by your dispatcher class in this CL)
> > - terminiating an existing worker (could also be handled by the dispatcher
> > class)
> > 
> > I guess what i'm getting at is that maybe we don't need to use the rounted
> > message infrastructure for this stuff. (Although, I see the devtools
> integration
> > done websharedworker_stub.cc might depend on it in some way?).
> 
> I expected having each worker's own routing might make code simpler, but I'm
> sure we can likely do the same without routing too.  What concerns you most is
> threading overhead btw?  I can pursue the model (no routing model) too.

Well, thinking this more avoiding to have the main worker messaging on main
thread sounds like a reasonable idea. Let me look into it deeper.

[michaeln, 2013-11-05 23:10:22]

> Sure, I had several more parameters for this but deleted them for now, as I
> wasn't sure what value would be exactly needed. (I should have left some TODO
> comments here)

Sure, that makes sense. In terms of the abstractions we talked about yesterday,
i think we need to know the ServiceWorkerVersion id.

[kinuko, 2013-11-06 14:36:02]

Revised the skeleton code, ended up adding a lot more code though it still has
some awkward part.
To help you review this code (or help you avoid reviewing the code) I've made a
rough sketch visualizing the overall structure:

https://docs.google.com/a/chromium.org/document/d/1HzjaXttZcRr6cE7oINMzvlVskb...

The blink side patch is here: https://codereview.chromium.org/61763004/

PTAL (code and/or design), thanks.

[kinuko, 2013-11-07 03:59:17]

Reviewing the design by myself, I think I should get rid of the worker:thread
map as there could still be a racy condition. I'll rework on it...

[kinuko, 2013-11-07 07:33:06]

Updated. 1) moved all embedded-worker related code into content/renderer for now
(as having it in content/child makes the patch slightly bloated) 2) dropped the
thread map.  PTAL

[michaeln, 2013-11-13 00:31:37]

This framing is looking pretty nice!

https://codereview.chromium.org/54573002/diff/1060001/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/1060001/content/common/service_...
content/common/service_worker_messages.h:64: IPC_STRUCT_MEMBER(string16,
script_source)
Like we've talked talked about, I think we want the embedded workercontext to be
responsible for loading it's main resource thru it's resource loaded. This
script_source param should not be needed.

https://codereview.chromium.org/54573002/diff/1060001/content/common/service_...
content/common/service_worker_messages.h:68:
IPC_STRUCT_MEMBER(blink::WebEmbeddedWorkerStartMode, start_mode)
what is "start_mode"?

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
File content/renderer/service_worker/embedded_worker_dispatcher.cc (right):

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
content/renderer/service_worker/embedded_worker_dispatcher.cc:102: // Posts a
delayed task to forcibly delete the worker instance to
Why do this?

I guess the goal is to invoke ChildProcess::current()->ReleaseProcess() in a
timely fashion. We might want to invoke a WebEmbeddedWorker::abort() method in
this delayed task distinguish between usual destruction sequence (where the
context has unwound and such) and this unusual sequence.

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
File content/renderer/service_worker/service_worker_context_client.cc (right):

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
content/renderer/service_worker/service_worker_context_client.cc:76: // messages
directly to this client.
i guess you mean messages for this likes of oninstall and onfetch

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
File content/renderer/service_worker/service_worker_context_message_filter.h
(right):

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
content/renderer/service_worker/service_worker_context_message_filter.h:12: //
Dispatches message to each ServiceWorker's worker context.
I see this cl depends on the CrossThreadMessageFilter (aka
ChildThreadMessageFilter).

[kinuko, 2013-11-18 14:09:48]

Updated, and removed some code that is not really wired yet. PTAL?

https://codereview.chromium.org/54573002/diff/1060001/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/1060001/content/common/service_...
content/common/service_worker_messages.h:64: IPC_STRUCT_MEMBER(string16,
script_source)
On 2013/11/13 00:31:37, michaeln wrote:
> Like we've talked talked about, I think we want the embedded workercontext to
be
> responsible for loading it's main resource thru it's resource loaded. This
> script_source param should not be needed.

Done.

https://codereview.chromium.org/54573002/diff/1060001/content/common/service_...
content/common/service_worker_messages.h:68:
IPC_STRUCT_MEMBER(blink::WebEmbeddedWorkerStartMode, start_mode)
On 2013/11/13 00:31:37, michaeln wrote:
> what is "start_mode"?

For devtools support. We can remove this for now (and I did so)

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
File content/renderer/service_worker/embedded_worker_dispatcher.cc (right):

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
content/renderer/service_worker/embedded_worker_dispatcher.cc:102: // Posts a
delayed task to forcibly delete the worker instance to
Yes, it's to make sure we terminate the worker context. Adding abort() sounds
good--- but let me postpone doing so and remove this part for now. (I wanted to
make sure we can add these hacks when necessary in this class structure too)

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
File content/renderer/service_worker/service_worker_context_client.cc (right):

https://codereview.chromium.org/54573002/diff/1060001/content/renderer/servic...
content/renderer/service_worker/service_worker_context_client.cc:76: // messages
directly to this client.
On 2013/11/13 00:31:37, michaeln wrote:
> i guess you mean messages for this likes of oninstall and onfetch

Yep.

[michaeln, 2013-11-19 01:17:07]

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
content/common/service_worker_messages.h:58: IPC_STRUCT_MEMBER(int32,
embedded_worker_id)
I have a question regarding int32 vs plain int? Our style guide says to prefer
plain 'int'. There are lots of <<>>_messages.h files that use plain 'int'.

Question is twofold...

Should we switch all of these messages to use plain int instead of int32 before
the 32 bit usages ripple far and wide?

Or has something changes such that we now prefer int32 in <<>>_message.h files? 
And if so, how do we prevent int32 types from rippling far and wide outside of
the message handlers?

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
content/common/service_worker_messages.h:61: IPC_STRUCT_MEMBER(string16,
content_security_policy)
Where do these values come from and what do they mean? Is the CSP and TYPE
something that is delivered with the main script resource (in a particular
header for example)?

[kinuko, 2013-11-19 01:49:10]

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
content/common/service_worker_messages.h:61: IPC_STRUCT_MEMBER(string16,
content_security_policy)
On 2013/11/19 01:17:08, michaeln wrote:
> Where do these values come from and what do they mean? Is the CSP and TYPE
> something that is delivered with the main script resource (in a particular
> header for example)?

That's what I'm trying to figure out now too..

As I wrote a question here:
https://codereview.chromium.org/70533005/diff/1/content/child/service_worker/...

Usually for shared / dedicated worker case we get the information from the
parent context's security context, but I'm not deadly sure yet how we should do
for service worker case (e.g. should we send it from provider host or if that's
something we could make up)

[kinuko, 2013-11-19 07:30:58]

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
content/common/service_worker_messages.h:61: IPC_STRUCT_MEMBER(string16,
content_security_policy)
On 2013/11/19 01:49:10, kinuko wrote:
> On 2013/11/19 01:17:08, michaeln wrote:
> > Where do these values come from and what do they mean? Is the CSP and TYPE
> > something that is delivered with the main script resource (in a particular
> > header for example)?
> 
> That's what I'm trying to figure out now too..
> 
> As I wrote a question here:
>
https://codereview.chromium.org/70533005/diff/1/content/child/service_worker/...
> 
> Usually for shared / dedicated worker case we get the information from the
> parent context's security context, but I'm not deadly sure yet how we should
do
> for service worker case (e.g. should we send it from provider host or if
that's
> something we could make up)

Some more info:

It's for CSP, and the string's coming from Content-security-policy HTTP header.
I think we probably need to send this string from SWProvider.

Nice doc by Mike West:
http://www.html5rocks.com/en/tutorials/security/content-security-policy/

[kinuko, 2013-11-19 07:45:41]

Changed int32's to int.

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
content/common/service_worker_messages.h:58: IPC_STRUCT_MEMBER(int32,
embedded_worker_id)
> Question is twofold...
> 
> Should we switch all of these messages to use plain int instead of int32
before
> the 32 bit usages ripple far and wide?

This sounds reasonable to me.
 
> Or has something changes such that we now prefer int32 in <<>>_message.h
files? 
> And if so, how do we prevent int32 types from rippling far and wide outside of
> the message handlers?

I don't think so, I basically blindly followed the convention in this file (and
also had a mindset that using specific int type in IPC is normal).

[alecflett, 2013-11-19 21:10:24]

On 2013/11/19 07:45:41, kinuko wrote:
> Changed int32's to int.
> 
>
https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
> File content/common/service_worker_messages.h (right):
> 
>
https://codereview.chromium.org/54573002/diff/1140002/content/common/service_...
> content/common/service_worker_messages.h:58: IPC_STRUCT_MEMBER(int32,
> embedded_worker_id)
> > Question is twofold...
> > 
> > Should we switch all of these messages to use plain int instead of int32
> before
> > the 32 bit usages ripple far and wide?
> 
> This sounds reasonable to me.
>  
> > Or has something changes such that we now prefer int32 in <<>>_message.h
> files? 
> > And if so, how do we prevent int32 types from rippling far and wide outside
of
> > the message handlers?
> 
> I don't think so, I basically blindly followed the convention in this file
(and
> also had a mindset that using specific int type in IPC is normal).

Sorry that was me - I was following convention in indexeddb which is obviously
in the minority here.. I had the same assumption about IPCs actually.

[michaeln, 2013-11-19 22:36:38]

Not sure what happened to my comment ^^^? Here it is again (or close to it).

Ah... thnx for the link to the CSP intro.

Given that the processing of ServiceWorkerMsg_StartWorker includes fetching the
main resource, including its headers, I don't think we need to pass in the
"Content-Security-Policy" header separately. That header value will be retrieved
when loading the main script resource.

[kinuko, 2013-11-20 01:37:07]

On 2013/11/19 22:36:38, michaeln wrote:
> Not sure what happened to my comment ^^^? Here it is again (or close to it).
> 
> Ah... thnx for the link to the CSP intro.
> 
> Given that the processing of ServiceWorkerMsg_StartWorker includes fetching
the
> main resource, including its headers, I don't think we need to pass in the
> "Content-Security-Policy" header separately. That header value will be
retrieved
> when loading the main script resource.

Can we revisit this later?  I can remove these params from IPC msg for now and
start with an empty string if this hurts your eyes in review.  My rough
impression is most of CSP logic is implemented in blink, and reusing the
existing logic would save some of our burden at least in an early stage (i.e. we
can revisit this later).

[michaeln, 2013-11-20 01:59:00]

> Can we revisit this later? I can remove these params from IPC msg for now and
> start with an empty string if this hurts your eyes in review.

Yes, please do.

I'm assuming the CSP header value we should be using is the CSP header value
that comes with the script_url response. Is that right?

[kinuko, 2013-11-20 04:00:55]

Removed the CSP params from IPC. (For now they're still in WorkerStartData in
blink API, but I'll revisit them later) PTAL?

[michaeln, 2013-11-20 23:03:47]

lgtm!

[kinuko, 2013-11-21 06:39:41]

(Removed CSP related code, waiting for the other blink patch to land)

[kinuko, 2013-11-21 10:13:09]

tsepez@: Can you review content/common/service_worker_messages.h changes?

jochen@: Can you do the owner review for content/renderer ?
(Mainly render_thread_impl.*)

[jochen (gone - plz use gerrit), 2013-11-21 12:21:31]

lgtm

[Tom Sepez, 2013-11-21 20:03:46]

messages LGTM with nits.

https://codereview.chromium.org/54573002/diff/2000001/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/2000001/content/common/service_...
content/common/service_worker_messages.h:11: #include
"third_party/WebKit/public/web/WebContentSecurityPolicy.h"
nit: is this include really necessary?

https://codereview.chromium.org/54573002/diff/2000001/content/common/service_...
content/common/service_worker_messages.h:59: int /* embedded_worker_id */,
nit: do we want this to be an int32 for consistency with the other types of _ids
we have in these messages?

[kinuko, 2013-11-22 03:46:19]

Thanks!

https://codereview.chromium.org/54573002/diff/2000001/content/common/service_...
File content/common/service_worker_messages.h (right):

https://codereview.chromium.org/54573002/diff/2000001/content/common/service_...
content/common/service_worker_messages.h:11: #include
"third_party/WebKit/public/web/WebContentSecurityPolicy.h"
On 2013/11/21 20:03:46, Tom Sepez wrote:
> nit: is this include really necessary?

Nope... removed.

https://codereview.chromium.org/54573002/diff/2000001/content/common/service_...
content/common/service_worker_messages.h:59: int /* embedded_worker_id */,
On 2013/11/21 20:03:46, Tom Sepez wrote:
> nit: do we want this to be an int32 for consistency with the other types of
_ids
> we have in these messages?

We're thinking about changing other int32's to int.

[commit-bot: I haz the power, 2013-11-28 04:43:06]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/kinuko@chromium.org/54573002/2170001

[commit-bot: I haz the power, 2013-11-28 04:57:58]

Retried try job too often on mac_rel for step(s) base_unittests, browser_tests,
interactive_ui_tests, net_unittests, unit_tests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=mac_rel&nu...

[commit-bot: I haz the power, 2013-11-29 01:49:22]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/kinuko@chromium.org/54573002/2170001

[commit-bot: I haz the power, 2013-11-29 02:23:17]

Retried try job too often on linux_rel for step(s) browser_tests, net_unittests,
unit_tests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=linux_rel&...

[commit-bot: I haz the power, 2013-11-29 03:31:23]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/kinuko@chromium.org/54573002/2170001

[commit-bot: I haz the power, 2013-11-29 04:35:04]

Change committed as 237873