Make it possible to configure synthetic delays through Devtools

Make it possible to configure synthetic delays through Devtools

Expose a method for configuring synthetic delays through the Devtools
tracing API. One message (Tracing.configureSyntheticDelays) is used to
configure the details, and another
(Tracing.configureSyntheticDelaysComplete) notifies that the operation
has completed.

BUG=307841

[Sami, 2013-12-02 17:37:03]

Here's how configuring synthetic delays could work through devtools. IMHO a
little cleaner than the render widget approach, but wdyt?

[tonyg, 2013-12-02 17:44:51]

On 2013/12/02 17:37:03, Sami wrote:
> Here's how configuring synthetic delays could work through devtools. IMHO a
> little cleaner than the render widget approach, but wdyt?

Teaching chrome to do synthetic delays seems like a flawed idea to me.
1. It is closely coupled with the implementation. So when code changes, the
delay code is not likely to be updated to keep pace. We are *really* bad about
updating things like this when we make architecture changes.
2. Seems really hard to ensure it is enforced across the board even with the
current code.
3. It is adding test code to the chrome binary, which ideally should be avoided.

For testing network performance, we have the same issue. Although you could
imagine adding synthetic delays in the network stack, instead we use dummynet to
hook in at the driver level to apply proper network simulation. Then whatever
happens in Chrome's architecture, we know our network delay simulation is still
valid.

Can we do something like that for the operations we are trying to delay here?
Something at the OS level instead of Chrome level?

[Sami, 2013-12-02 18:34:14]

On 2013/12/02 17:44:51, tonyg wrote:
> Teaching chrome to do synthetic delays seems like a flawed idea to me.
> 1. It is closely coupled with the implementation. So when code changes, the
> delay code is not likely to be updated to keep pace. We are *really* bad about
> updating things like this when we make architecture changes.
> 2. Seems really hard to ensure it is enforced across the board even with the
> current code.
> 3. It is adding test code to the chrome binary, which ideally should be
avoided.
> 
> For testing network performance, we have the same issue. Although you could
> imagine adding synthetic delays in the network stack, instead we use dummynet
to
> hook in at the driver level to apply proper network simulation. Then whatever
> happens in Chrome's architecture, we know our network delay simulation is
still
> valid.
> 
> Can we do something like that for the operations we are trying to delay here?
> Something at the OS level instead of Chrome level?

I've also had similar concerns to your points above, which is why I've kept
trying to come up with alternate ways to test the scenarios from the scheduling
test suite (https://codereview.chromium.org/68203031/). Unfortunately in general
the systems that introduce latency to Chrome's input processing and rendering
aren't as readily isolated as the networking stack, so I haven't found a silver
bullet yet. Also, trying to provoke these delays just with different JavaScript
or DOM workloads is very hard to do consistently across platforms.

That said, now that I've got a baseline to calibrate against, I could have
another go at less invasive methods for testing some of these things. Especially
the graphics part should be easy to isolate with an LD_PRELOADed shim. I've also
been thinking of throttling the CPU time of the different threads in Chrome as a
proxy for high workloads.

[tonyg, 2013-12-02 18:49:20]

On 2013/12/02 18:34:14, Sami wrote:
> On 2013/12/02 17:44:51, tonyg wrote:
> > Teaching chrome to do synthetic delays seems like a flawed idea to me.
> > 1. It is closely coupled with the implementation. So when code changes, the
> > delay code is not likely to be updated to keep pace. We are *really* bad
about
> > updating things like this when we make architecture changes.
> > 2. Seems really hard to ensure it is enforced across the board even with the
> > current code.
> > 3. It is adding test code to the chrome binary, which ideally should be
> avoided.
> > 
> > For testing network performance, we have the same issue. Although you could
> > imagine adding synthetic delays in the network stack, instead we use
dummynet
> to
> > hook in at the driver level to apply proper network simulation. Then
whatever
> > happens in Chrome's architecture, we know our network delay simulation is
> still
> > valid.
> > 
> > Can we do something like that for the operations we are trying to delay
here?
> > Something at the OS level instead of Chrome level?
> 
> I've also had similar concerns to your points above, which is why I've kept
> trying to come up with alternate ways to test the scenarios from the
scheduling
> test suite (https://codereview.chromium.org/68203031/). Unfortunately in
general
> the systems that introduce latency to Chrome's input processing and rendering
> aren't as readily isolated as the networking stack, so I haven't found a
silver
> bullet yet. Also, trying to provoke these delays just with different
JavaScript
> or DOM workloads is very hard to do consistently across platforms.
> 
> That said, now that I've got a baseline to calibrate against, I could have
> another go at less invasive methods for testing some of these things.
Especially
> the graphics part should be easy to isolate with an LD_PRELOADed shim. I've
also
> been thinking of throttling the CPU time of the different threads in Chrome as
a
> proxy for high workloads.

Cool, good to hear. I've also wondered whether linux's fio command might be
useful to us for things like this. Never really tested it properly though.

[Sami, 2013-12-02 18:59:26]

On 2013/12/02 18:49:20, tonyg wrote:
> Cool, good to hear. I've also wondered whether linux's fio command might be
> useful to us for things like this. Never really tested it properly though.

Did you mean running fio in parallel to the actual benchmark in Chrome? I'm not
very familiar with it either. I wonder if it's purely for IO workloads.

[nduca, 2013-12-02 22:06:47]

TBH I'm quite skeptical of this os-level hooking approach. I understand why you
suggest it, but I think there's an impedance mismatch: whereas much of the
delays in the net stack come from os level delays, much of our delays come from
non-os things that aren't obviously shimmable. Also, having to do shimming means
having to take every place we see a delay and finding a way to shim it 5
different times. We have a half dozen delay points, even in v0, so we're talking
some real work I think.

Sami, what about expanding categoryFilter to understand cc.debug,cc.thingDelay=7
 as enable cc.debug and set cc.thingDelay to 7. Then you dont have any surface
area added to the tracing apis but you still allow programming of the delay
system.

[Sami, 2013-12-03 17:45:54]

On 2013/12/02 22:06:47, nduca wrote:
> Sami, what about expanding categoryFilter to understand
cc.debug,cc.thingDelay=7
>  as enable cc.debug and set cc.thingDelay to 7. Then you dont have any surface
> area added to the tracing apis but you still allow programming of the delay
> system.

I've now got something along those lines in
https://codereview.chromium.org/98953002/. Lemme know what you guys think.

[tonyg, 2013-12-04 16:45:02]

On 2013/12/02 22:06:47, nduca wrote:
> TBH I'm quite skeptical of this os-level hooking approach. I understand why
you
> suggest it, but I think there's an impedance mismatch: whereas much of the
> delays in the net stack come from os level delays, much of our delays come
from
> non-os things that aren't obviously shimmable. Also, having to do shimming
means
> having to take every place we see a delay and finding a way to shim it 5
> different times. We have a half dozen delay points, even in v0, so we're
talking
> some real work I think.

I'd like to swing by your desk and understand this a bit better.

Even ignoring the bittleness/maintenance concern I mentioned, delaying trace
events seems like a misleading method to simulate whatever slow condition we are
actually trying to simulate (eg. slow IO, slow disk, slow memory bandwidth, slow
GL calls, slow CPU, etc. etc.). The problem is that it doesn't simulate
contention properly. A slow event probably is introducing contention on one of
the system that we are really trying to simulate. But if we just sleep during
the trace event it makes it look like that resource is idle and free to use. So
the tempting solution would be to stack on the parallelism, potentially making
the benchmark better and the real-users worse.

Maybe we can get some better ideas if we talk more specifically about which
subsystem you are trying to simulate slowness within?

[tonyg, 2013-12-04 16:45:03]

On 2013/12/02 22:06:47, nduca wrote:
> TBH I'm quite skeptical of this os-level hooking approach. I understand why
you
> suggest it, but I think there's an impedance mismatch: whereas much of the
> delays in the net stack come from os level delays, much of our delays come
from
> non-os things that aren't obviously shimmable. Also, having to do shimming
means
> having to take every place we see a delay and finding a way to shim it 5
> different times. We have a half dozen delay points, even in v0, so we're
talking
> some real work I think.

I'd like to swing by your desk and understand this a bit better.

Even ignoring the bittleness/maintenance concern I mentioned, delaying trace
events seems like a misleading method to simulate whatever slow condition we are
actually trying to simulate (eg. slow IO, slow disk, slow memory bandwidth, slow
GL calls, slow CPU, etc. etc.). The problem is that it doesn't simulate
contention properly. A slow event probably is introducing contention on one of
the system that we are really trying to simulate. But if we just sleep during
the trace event it makes it look like that resource is idle and free to use. So
the tempting solution would be to stack on the parallelism, potentially making
the benchmark better and the real-users worse.

Maybe we can get some better ideas if we talk more specifically about which
subsystem you are trying to simulate slowness within?

[Sami, 2013-12-04 16:53:20]

On 2013/12/04 16:45:03, tonyg wrote:
> On 2013/12/02 22:06:47, nduca wrote:
> > TBH I'm quite skeptical of this os-level hooking approach. I understand why
> you
> > suggest it, but I think there's an impedance mismatch: whereas much of the
> > delays in the net stack come from os level delays, much of our delays come
> from
> > non-os things that aren't obviously shimmable. Also, having to do shimming
> means
> > having to take every place we see a delay and finding a way to shim it 5
> > different times. We have a half dozen delay points, even in v0, so we're
> talking
> > some real work I think.
> 
> I'd like to swing by your desk and understand this a bit better.

Please keep me posted and/or grab me for a vc :)
 
> Even ignoring the bittleness/maintenance concern I mentioned, delaying trace
> events seems like a misleading method to simulate whatever slow condition we
are
> actually trying to simulate (eg. slow IO, slow disk, slow memory bandwidth,
slow
> GL calls, slow CPU, etc. etc.). The problem is that it doesn't simulate
> contention properly. A slow event probably is introducing contention on one of
> the system that we are really trying to simulate. But if we just sleep during
> the trace event it makes it look like that resource is idle and free to use.
So
> the tempting solution would be to stack on the parallelism, potentially making
> the benchmark better and the real-users worse.
> 
> Maybe we can get some better ideas if we talk more specifically about which
> subsystem you are trying to simulate slowness within?

For reference, here's the current set of delay points (from
https://codereview.chromium.org/83183005/)

  1. blink.HandleInputEvent - Expensive JavaScript input event handling.
  2. cc.RasterRequiredForActivation - Long running raster tasks.
  3. cc.BeginMainFrame - Complex main thread picture recording, layout or rAF.
  4. gpu.AsyncTexImage - Slow texture uploads.
  5. gpu.SwapBuffers - GPU-bound rendering.

This is the design doc in case you haven't seen it:
https://docs.google.com/document/d/17yhE5Po9By0sCdM1yZT3LiUECaUr_94rQt9j-4tOQ...

The current implementation uses a busy loop to simulate contention, which should
be more realistic than sleeping. We could also add some load on the memory bus.

[Sami, 2013-12-09 15:48:14]

> On 2013/12/04 16:45:03, tonyg wrote:
> > I'd like to swing by your desk and understand this a bit better.

Hi guys, any resolution here? Anything I can do to help?

To give an update on the alternative approach I had in mind, we can easily
throttle GL functions with an LD_PRELOAD shim, but since the number of
operations changes between devices (because of different viewport sizes) it's
not easy to apply a consistent delay. I also tried throttling specific threads
in Chrome using SIGSTOP/SIGCONT with tgkill(), but somehow that was still
suspending the whole process. I also feel that the thread granularity of this
approach may be too coarse. I also considered detours-style code injection
(maybe through gdb) but it'll take a lot of effort to get that going on even a
single platform.

[tonyg_google, 2013-12-09 16:22:58]

Nat and I caught up on this. tldr; is that he convinced me there is a
use case for this, but we both agreed there is a potential for
foot-gunning. Sami and I are going to chat about it today and then
we'll post a summary to the thread.

On Mon, Dec 9, 2013 at 7:48 AM,  <skyostil@chromium.org> wrote:
>> On 2013/12/04 16:45:03, tonyg wrote:
>> > I'd like to swing by your desk and understand this a bit better.
>
>
> Hi guys, any resolution here? Anything I can do to help?
>
> To give an update on the alternative approach I had in mind, we can easily
> throttle GL functions with an LD_PRELOAD shim, but since the number of
> operations changes between devices (because of different viewport sizes)
> it's
> not easy to apply a consistent delay. I also tried throttling specific
> threads
> in Chrome using SIGSTOP/SIGCONT with tgkill(), but somehow that was still
> suspending the whole process. I also feel that the thread granularity of
> this
> approach may be too coarse. I also considered detours-style code injection
> (maybe through gdb) but it'll take a lot of effort to get that going on even
> a
> single platform.
>
> https://codereview.chromium.org/99543002/

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[Sami, 2013-12-09 19:17:08]

On 2013/12/09 16:22:58, tonyg_google wrote:
> Nat and I caught up on this. tldr; is that he convinced me there is a
> use case for this, but we both agreed there is a potential for
> foot-gunning. Sami and I are going to chat about it today and then
> we'll post a summary to the thread.

Thanks for the chat Tony. My main takeaway was that we should only use synthetic
delays in (micro)benchmarks where they truly make sense. The danger is that we'd
have them all over the place and then made optimizations that improve those
numbers but wouldn't actually have a positive effect in the real world.

It seems like the right way forward is to use delay-based testing for specific
components that require it (e.g., scheduler) and to keep developing more
implementation-agnostic end-to-end tests for things like smoothness and latency.
This combination should serve as a robust guard against regressions and a guide
for optimization and larger architecture changes.