[Remoting] OpenGL Rendering Layer

remoting/client/opengl/gl_desktop.cc

Index: remoting/client/opengl/gl_desktop.cc
diff --git a/remoting/client/opengl/gl_desktop.cc b/remoting/client/opengl/gl_desktop.cc
new file mode 100644
index 0000000000000000000000000000000000000000..e1587e95875ee22e20eda942e8d7ffe4401cff04
--- /dev/null
+++ b/remoting/client/opengl/gl_desktop.cc
@@ -0,0 +1,49 @@
+// Copyright 2016 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include "remoting/client/opengl/gl_desktop.h"
+
+#include "base/logging.h"
+#include "remoting/client/opengl/gl_canvas.h"
+#include "remoting/client/opengl/gl_render_layer.h"
+#include "third_party/webrtc/modules/desktop_capture/desktop_frame.h"
+
+namespace {
+
+const int kTextureId = 0;
+}
+
+namespace remoting {
+
+GlDesktop::GlDesktop() {}
+
+GlDesktop::~GlDesktop() {}
+
+void GlDesktop::SetCanvas(GlCanvas* canvas) {
+  if (!canvas) {
+    layer_.reset();
+    return;
+  }
+  layer_.reset(new GlRenderLayer(kTextureId, canvas));
+  if (last_frame_) {
+    layer_->SetTexture(last_frame_->data(), last_frame_->size().width(),
+                       last_frame_->size().height());
+  }
+}
+
+void GlDesktop::SetVideoFrame(std::unique_ptr<webrtc::DesktopFrame> frame) {
+  if (layer_) {
+    layer_->SetTexture(frame->data(), frame->size().width(),

[Sergey Ulanov, 2016/06/27 23:15:29]

It's best to avoid updating the whole texture every time. frame->update_region()
contains information about parts of the screen that have changed since the
previous frame.

[Sergey Ulanov, 2016/06/27 23:15:29]

Here you assume that the the frame lines are packed in DesktopFrame next to each
other. That may not be true. There is GL_UNPACK_ROW_LENGTH attribute you need to
set when calling glTexImage3D() to make it use correct stride.

[Yuwei, 2016/06/28 01:31:20]

By using SoftwareVideoRenderer and SharedDesktopFrame it seems the lines are
always packed next to each other...

I had considered glPixelStorei(GL_UNPACK_ROW_LENGTH,stride) and only update
changed regions but turns out GL_UNPACK_ROW_LENGTH isn't even supported in
OpenGL ES 2. In this case we will have to upload texture line by line...

Not sure whether we can use OpenGL ES 3 which supports less devices. It supports
pixel buffer and we can draw the desktop directly onto the buffer.

[Yuwei, 2016/06/29 17:36:22]

Turns out OpenGL ES 3 is backward compatible with ES 2 and you can decide which
OpenGL client version should be used when creating the EGL context.

Maybe implement PBO texture update for ES 3 and fall back to either updating row
by row or constructing the full image in CPU when ES 3 is not supported?

[Yuwei, 2016/06/30 00:04:55]

Just measured some render latencies when playing YouTube:

* Use SharedDesktopFrame, updating entire frame, no lock, average 9ms.
* Use BasicDesktopFrame & ES 3, update using GL_UNPACK_ROW_LENGTH, average 8ms.
* Use BasicDesktopFrame & ES 2, update row by row, average 59ms :(

I think using SharedDesktopFrame is mostly good enough? It is quite annoying
that ES 2 doesn't support row length. I'll also take a look at wrapping PBO into
DesktopFrame.

Just have some questions:

* Will SharedDesktopFrame be always tightly packed, and in what situation it
won't? Is this host dependent?
* Is it good to use SharedDesktopFrame without a lock? I'm not touching anything
like dirty region so the only concern is sometimes you may see a frame being
half-updated...

[Sergey Ulanov, 2016/06/30 00:34:52]

Is this when updating the whole frame? I think the difference between these two
would be much larger for partial frame updates.
With ES2 it's probably better to copy all changed rows into a tightly packed
buffer and update the texture from there, repeat for each updated rect. That
should be faster for partial updates.
Are you suggesting that we use a single frame buffer and keep using it for every
frame?
If yes, then this should work, but it will be slower for partial updates.
It's determined by the frame type inside SharedDesktopFrame. BasicDesktopFrame()
is always packed. So yes, we can make sure that it's always packed as long as we
use BasicDesktopFrame().
Correct, as long as you own the SharedDesktopFrame object. SharedDesktopFrame
instances share only the underlying buffer. This may cause artifacts when
decoding to the same buffer on a different thread, but no issues otherwise.

[Yuwei, 2016/07/07 22:07:45]

Done.

+                       frame->size().height());
+  }
+  last_frame_ = std::move(frame);
+}
+
+void GlDesktop::Draw() {
+  if (layer_) {
+    layer_->Draw();
+  }
+}
+
+}  // namespace remoting