Add scroll perf test for protocol in chromoting

Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Committed: https://crrev.com/11695ba56f1d7843ca1e083d9f0d60145ad03c4b
Cr-Original-Commit-Position: refs/heads/master@{#407957}
Cr-Commit-Position: refs/heads/master@{#408189}

[Sergey Ulanov, 2016-07-26 00:21:35]

Patchset #1 (id:1) has been deleted

[Sergey Ulanov, 2016-07-26 00:21:46]

sergeyu@chromium.org changed reviewers:
+ isheriff@chromium.org

[Irfan, 2016-07-26 16:45:58]

I am curious if the latency numbers provide any useful information over previous
perf tests. Can you put it in the commit message ? 

Probably also worth commenting the difference b/w scroll test and perf test
somewhere in there and the value scroll test adds.

https://codereview.chromium.org/2182833002/diff/20001/remoting/test/scroll_fr...
File remoting/test/scroll_frame_generator.h (right):

https://codereview.chromium.org/2182833002/diff/20001/remoting/test/scroll_fr...
remoting/test/scroll_frame_generator.h:35: std::map<int, base::TimeTicks>
frame_timestamp_;
unordered_map since we do not need traversal ?

[Sergey Ulanov, 2016-07-26 19:56:30]

Description was changed from

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.
==========

to

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0:
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1:
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915
==========

[Sergey Ulanov, 2016-07-26 21:47:18]

Added more comments and moved some test impl in the file to make it clear how
the parts are related to each other.

Also added some numbers I get with the new test in the CL description. My
current observations:
1. WebRTC performs worse with high bandwidth: frame rate is much lower, around
10 fps. I didn't dig deep into why this happens, but I think it's largely due to
the way we currently schedule captures: bandwidth is wasted by not sending
anything while each frame is being encoded. It shouldn't be hard to improve
behavior in this scenario.

2. WebRTC performs much better in case when bandwidth is limited (much lower
latency). This can be explained by different encoder setting we use for WebRTC
and the way frame capture is scheduled. In this scenario old protocol always
keeps 4 frames in the send queue, each taking a lot of time to send.

In other words old protocol perform well when the bandwidth is high enough to
push close to 30fps, but degrades greatly with lower bandwidth. This is not
surprising given that it doesn't adapt codec settings to match bandwidth.

Currently the test also doesn't quantify video quality in any way. E.g. we need
to make sure that we can reach the same quality as with the old protocol. I'm
looking into how that can be fixed.

https://codereview.chromium.org/2182833002/diff/20001/remoting/test/scroll_fr...
File remoting/test/scroll_frame_generator.h (right):

https://codereview.chromium.org/2182833002/diff/20001/remoting/test/scroll_fr...
remoting/test/scroll_frame_generator.h:35: std::map<int, base::TimeTicks>
frame_timestamp_;
On 2016/07/26 16:45:58, Irfan wrote:
> unordered_map since we do not need traversal ?

Done.

[Sergey Ulanov, 2016-07-26 21:55:07]

Description was changed from

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0:
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1:
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915
==========

to

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915
==========

[Sergey Ulanov, 2016-07-26 21:57:39]

Also updated configuration for LimitedBandwidth. Previously bandwidth numbers
didn't match the comments.

[Irfan, 2016-07-26 22:01:02]

lgtm

[Sergey Ulanov, 2016-07-26 22:09:36]

The CQ bit was checked by sergeyu@chromium.org

[commit-bot: I haz the power, 2016-07-26 22:10:35]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-07-26 23:11:11]

Description was changed from

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915
==========

to

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915
==========

[commit-bot: I haz the power, 2016-07-26 23:11:12]

Committed patchset #3 (id:60001)

[commit-bot: I haz the power, 2016-07-26 23:14:42]

Description was changed from

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915
==========

to

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Cr-Commit-Position: refs/heads/master@{#407957}
==========

[commit-bot: I haz the power, 2016-07-26 23:14:47]

Patchset 3 (id:??) landed as
https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Cr-Commit-Position: refs/heads/master@{#407957}

[robliao, 2016-07-27 00:33:43]

A revert of this CL (patchset #3 id:60001) has been created in
https://codereview.chromium.org/2190473002/ by robliao@chromium.org.

The reason for reverting is: Causes build errors

https://uberchromegw.corp.google.com/i/chromium.win/builders/Win8%20GYP%20%28...

[4615/7755] LINK_EMBED remoting_perftests.exe
FAILED: remoting_perftests.exe remoting_perftests.exe.pdb 
E:\b\depot_tools\python276_bin\python.exe gyp-win-tool link-with-manifests
environment.x86 True remoting_perftests.exe
"E:\b\depot_tools\python276_bin\python.exe gyp-win-tool link-wrapper
environment.x86 False link.exe /nologo /OUT:remoting_perftests.exe
@remoting_perftests.exe.rsp" 1 mt.exe rc.exe
"obj\remoting\remoting_perftests.remoting_perftests.exe.intermediate.manifest"
obj\remoting\remoting_perftests.remoting_perftests.exe.generated.manifest
..\..\build\win\compatibility.manifest
remoting_perftests.protocol_perftest.obj : error LNK2019: unresolved external
symbol "public: __thiscall
remoting::test::ScrollFrameGenerator::ScrollFrameGenerator(void)"
(??0ScrollFrameGenerator@test@remoting@@QAE@XZ) referenced in function
"protected: void __thiscall
remoting::ProtocolPerfTest::MeasureScrollPerformance(bool)"
(?MeasureScrollPerformance@ProtocolPerfTest@remoting@@IAEX_N@Z)

remoting_perftests.protocol_perftest.obj : error LNK2019: unresolved external
symbol "public: class std::unique_ptr<class webrtc::DesktopFrame,struct
std::default_delete<class webrtc::DesktopFrame> > __thiscall
remoting::test::ScrollFrameGenerator::GenerateFrame(class
webrtc::SharedMemoryFactory *)"
(?GenerateFrame@ScrollFrameGenerator@test@remoting@@QAE?AV?$unique_ptr@VDesktopFrame@webrtc@@U?$default_delete@VDesktopFrame@webrtc@@@std@@@std@@PAVSharedMemoryFactory@webrtc@@@Z)
referenced in function "protected: void __thiscall
remoting::ProtocolPerfTest::MeasureScrollPerformance(bool)"
(?MeasureScrollPerformance@ProtocolPerfTest@remoting@@IAEX_N@Z)

remoting_perftests.protocol_perftest.obj : error LNK2019: unresolved external
symbol "public: class base::TimeDelta __thiscall
remoting::test::ScrollFrameGenerator::GetFrameLatency(class webrtc::DesktopFrame
const &)"
(?GetFrameLatency@ScrollFrameGenerator@test@remoting@@QAE?AVTimeDelta@base@@ABVDesktopFrame@webrtc@@@Z)
referenced in function "protected: void __thiscall
remoting::ProtocolPerfTest::MeasureScrollPerformance(bool)"
(?MeasureScrollPerformance@ProtocolPerfTest@remoting@@IAEX_N@Z)

remoting_perftests.protocol_perftest.obj : error LNK2019: unresolved external
symbol "private: __thiscall
remoting::test::ScrollFrameGenerator::~ScrollFrameGenerator(void)"
(??1ScrollFrameGenerator@test@remoting@@AAE@XZ) referenced in function "private:
void * __thiscall remoting::test::ScrollFrameGenerator::`scalar deleting
destructor'(unsigned int)" (??_GScrollFrameGenerator@test@remoting@@AAEPAXI@Z)

remoting_perftests.exe : fatal error LNK1120: 4 unresolved externals.

[Sergey Ulanov, 2016-07-27 17:29:23]

Description was changed from

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Cr-Commit-Position: refs/heads/master@{#407957}
==========

to

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Cr-Commit-Position: refs/heads/master@{#407957}
==========

[Sergey Ulanov, 2016-07-27 17:31:02]

The CQ bit was checked by sergeyu@chromium.org

[Sergey Ulanov, 2016-07-27 17:31:03]

The patchset sent to the CQ was uploaded after l-g-t-m from
isheriff@chromium.org
Link to the patchset: https://codereview.chromium.org/2182833002/#ps80001
(title: "gyp")

[commit-bot: I haz the power, 2016-07-27 17:31:26]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-07-27 18:27:02]

Description was changed from

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Cr-Commit-Position: refs/heads/master@{#407957}
==========

to

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Cr-Commit-Position: refs/heads/master@{#407957}
==========

[commit-bot: I haz the power, 2016-07-27 18:27:02]

Committed patchset #4 (id:80001)

[commit-bot: I haz the power, 2016-07-27 18:28:23]

Description was changed from

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Cr-Commit-Position: refs/heads/master@{#407957}
==========

to

==========
Add scroll perf test for protocol in chromoting

The new test simulates frame scroll and measures frame-rate and average
latency.

Some results from the new test:

LimitedBandwidth/0 (100Mbps):
ICE: FPS: 29.7853, Average latency: 49.7429
Webrtc: FPS: 10.4109, Average latency: 71.0577

LimitedBandwidth/1 (8Mbps):
ICE: FPS: 5.24054, Average latency: 734.131
Webrtc: FPS: 11.1471, Average latency: 116.915

Committed: https://crrev.com/8bc24e1f6de205377e3dbb814aeecaf8fe05a867
Committed: https://crrev.com/11695ba56f1d7843ca1e083d9f0d60145ad03c4b
Cr-Original-Commit-Position: refs/heads/master@{#407957}
Cr-Commit-Position: refs/heads/master@{#408189}
==========

[commit-bot: I haz the power, 2016-07-27 18:28:24]

Patchset 4 (id:??) landed as
https://crrev.com/11695ba56f1d7843ca1e083d9f0d60145ad03c4b
Cr-Commit-Position: refs/heads/master@{#408189}