Windows Host Touch Injection

remoting/host/touch_injector_win.cc

Index: remoting/host/touch_injector_win.cc
diff --git a/remoting/host/touch_injector_win.cc b/remoting/host/touch_injector_win.cc
new file mode 100644
index 0000000000000000000000000000000000000000..ebd370f38e3b7d5cc92ba9f92170e0c1b3ee316a
--- /dev/null
+++ b/remoting/host/touch_injector_win.cc
@@ -0,0 +1,284 @@
+// Copyright 2015 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include "remoting/host/touch_injector_win.h"
+
+#include "base/logging.h"
+#include "base/stl_util.h"
+#include "remoting/proto/event.pb.h"
+
+namespace remoting {
+
+using protocol::TouchEvent;
+using protocol::TouchEventPoint;
+
+namespace {
+
+typedef BOOL(NTAPI* InitializeTouchInjectionFunction)(UINT32, DWORD);
+typedef BOOL(NTAPI* InjectTouchInputFunction)(UINT32,
+                                              const POINTER_TOUCH_INFO*);
+const uint32_t kMaxSimlutaneousTouchCount = 10;

[Wez, 2015/03/17 03:06:23]

typo: Simultaneous

[Rintaro Kuroiwa, 2015/03/18 01:33:30]

Done.

+
+// Change the POINTER_TOUCH_INFO structures in the map to MOVE and copy those
+// into |output_vector|.

[Wez, 2015/03/17 03:06:23]

Why are you changing them in the map?

Isn't the point here just to pre-populate the vector<> with MOVE info for each
of the currently-down touch points?

It feels like you should be able to have a single function that takes whatever
is in the touches map and injects it, and then each of of the
add/move/end/cancel functions just updates the map, e.g. marking some points as
cancelled, does the injection, and then does any necessary cleanup, e.g.
removing cancelled points.  Potentially slightly less efficient, but easier to
read!

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

Done.
As discussed offline, the map will always contain POINTER_TOUCH_INFO as "move"
points. So this function is renamed.

+// This is used to reinject all points that have not changed as "move"d point,
+// even if it has not moved.
+// This is required for multi-touch to work, e.g. pinching and zooming gestures
+// (handled by apps) won't work without reinjection the points. If not
+// reinjected, the injection API assumes that they were dropped.
+void ChangeToMovePointsAndAppendToVector(
+    std::map<uint32_t, POINTER_TOUCH_INFO>* touches_in_contact,
+    std::vector<POINTER_TOUCH_INFO>* output_vector) {
+  DCHECK(touches_in_contact);
+  DCHECK(output_vector);

[Wez, 2015/03/17 03:06:23]

You are about to dereference these anyway, below, which will crash if they're
NULL, so it's not really worthwhile DCHECKing them first here.

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

Done.

+  for (auto& id_and_pointer_touch_info : *touches_in_contact) {
+    POINTER_TOUCH_INFO& pointer_touch_info = id_and_pointer_touch_info.second;
+    pointer_touch_info.pointerInfo.pointerFlags =
+        POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT | POINTER_FLAG_UPDATE;
+    output_vector->push_back(pointer_touch_info);
+  }
+}
+
+// The caller should set pointer_touch_info->pointerInfo.pointerFlags.
+void ConvertToPointerTouchInfo(
+    const TouchEventPoint& touch_point,
+    POINTER_TOUCH_INFO* pointer_touch_info) {
+  DCHECK(pointer_touch_info);

[Wez, 2015/03/17 03:06:23]

Same here

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

Done.

+
+  pointer_touch_info->touchMask =
+      TOUCH_MASK_CONTACTAREA | TOUCH_MASK_ORIENTATION;
+  pointer_touch_info->touchFlags = TOUCH_FLAG_NONE;
+
+  // Although radius_{x,y} can be undefined (i.e. has_radius_{x,y} == false),
+  // the default value will set the area correctly.

[Wez, 2015/03/17 03:06:23]

Why will the default value set it correctly? That relies on radius_{x,y} being
zero if un-set, and that the Windows API treats that as "default", right?

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

Yes it does rely on get on unset values return 0.
According to MSDN if the digitizer does not report the size then rcContact
should be 0 by 0 (size) centered at the touch coordinate.
I'll add a comment here.

+  pointer_touch_info->rcContact.left =
+      touch_point.x() - touch_point.radius_x();
+  pointer_touch_info->rcContact.top = touch_point.y() - touch_point.radius_y();
+  pointer_touch_info->rcContact.right =
+      touch_point.x() + touch_point.radius_x();
+  pointer_touch_info->rcContact.bottom =
+      touch_point.y() + touch_point.radius_y();
+
+  pointer_touch_info->orientation = touch_point.angle();
+
+  if (touch_point.has_pressure()) {
+    pointer_touch_info->touchMask |= TOUCH_MASK_PRESSURE;
+    const int kMaxTouchPressure = 1024;  // Defined in MSDN.
+    const int pressure = touch_point.pressure() * kMaxTouchPressure;

[Wez, 2015/03/17 03:06:23]

Is there a danger of this wrapping? Should we instead range-check
touch_point.pressure() before multiplying?

Should we bother clamping out-of-range pressures, or just ignore events with
bogus values in them?

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

Done.

+    if (pressure < 0) {
+      pointer_touch_info->pressure = 0;
+    } else if (pressure > kMaxTouchPressure) {
+      pointer_touch_info->pressure = kMaxTouchPressure;
+    } else {
+      pointer_touch_info->pressure = pressure;
+    }
+  } else {
+    // For a weird case where the pressure was there for a start event but
+    // not set in later events, set pressure to 0 because usually
+    // POINTER_TOUCH_INFO is memset() to 0.

[Wez, 2015/03/17 03:06:23]

Is there a reason you don't both memset()ting it?

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

I was thinking if I memset(0) that's an extra write. Probably not a big
performance problem so I'll memset.

+    pointer_touch_info->pressure = 0;
+  }
+
+  pointer_touch_info->pointerInfo.pointerType = PT_TOUCH;
+  pointer_touch_info->pointerInfo.pointerId = touch_point.id();
+  pointer_touch_info->pointerInfo.ptPixelLocation.x = touch_point.x();
+  pointer_touch_info->pointerInfo.ptPixelLocation.y = touch_point.y();
+}
+
+}  // namespace
+
+TouchInjectFunctionsWin::TouchInjectFunctionsWin()
+    : initialize_touch_injection_func_(nullptr),
+      inject_touch_input_func_(nullptr) {}
+
+TouchInjectFunctionsWin::~TouchInjectFunctionsWin() {}
+
+bool TouchInjectFunctionsWin::Init() {

[Wez, 2015/03/17 03:06:23]

You can use base::NativeLibrary, which wraps all this up in a cleaner way with
proper library ref-counting.

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

Done.

+  if (initialize_touch_injection_func_ && inject_touch_input_func_)
+    return true;
+
+  initialize_touch_injection_func_ = nullptr;
+  inject_touch_input_func_ = nullptr;

[Wez, 2015/03/17 03:06:23]

Why are you NULLing these? You just checked that they're null, and they're just
bare pointers, and the code below can't set one without setting the other.

[Rintaro Kuroiwa, 2015/03/18 01:33:30]

N/A in the new patch.

+  HMODULE user32_dll_handle = ::GetModuleHandleA("User32.dll");
+  if (!user32_dll_handle) {
+    PLOG(INFO) << "Failed to get User32.dll handle.";
+    return false;
+  }
+
+  InitializeTouchInjectionFunction init_func =
+       reinterpret_cast<InitializeTouchInjectionFunction>(
+           ::GetProcAddress(user32_dll_handle, "InitializeTouchInjection"));
+  if (!init_func) {
+    PLOG(INFO) << "Failed to get InitializeTouchInjection function handle.";
+    return false;
+  }
+
+  InjectTouchInputFunction inject_touch_func =
+      reinterpret_cast<InjectTouchInputFunction>(
+          ::GetProcAddress(user32_dll_handle, "InjectTouchInput"));
+  if (!inject_touch_func) {
+    PLOG(INFO) << "Failed to get InjectTouchInput.";
+    return false;
+  }
+
+  initialize_touch_injection_func_ = init_func;
+  inject_touch_input_func_ = inject_touch_func;
+  return true;
+}
+
+bool TouchInjectFunctionsWin::Initialized() {
+  DCHECK((initialize_touch_injection_func_ && inject_touch_input_func_) ||
+         (!initialize_touch_injection_func_ && !inject_touch_input_func_));
+  return initialize_touch_injection_func_ != nullptr;
+}
+
+BOOL TouchInjectFunctionsWin::InitializeTouchInjection(UINT32 max_count,
+                                                      DWORD dw_mode) {

[Wez, 2015/03/17 03:06:23]

Indentation

[Rintaro Kuroiwa, 2015/03/18 01:33:31]

Done.

+  return initialize_touch_injection_func_(max_count, dw_mode);
+}
+
+DWORD TouchInjectFunctionsWin::InjectTouchInput(
+    UINT32 count,
+    const POINTER_TOUCH_INFO* contacts) {
+  return inject_touch_input_func_(count, contacts);
+}
+
+TouchInjectorWin::TouchInjectorWin()
+    : delegate_(new TouchInjectFunctionsWin()) {}
+
+TouchInjectorWin::~TouchInjectorWin() {}
+
+bool TouchInjectorWin::Init() {
+  if (!delegate_->Init())
+    return false;
+
+  if (!delegate_->InitializeTouchInjection(kMaxSimlutaneousTouchCount,
+                                           TOUCH_FEEDBACK_DEFAULT)) {
+    PLOG(INFO) << "Failed to initialize touch injection.";

[Wez, 2015/03/17 03:06:23]

If you null the delegate_ here then you can check |delegate_| to determine
whether injection will work, and you don't need an Initialized() checker on the
delegate then.

[Rintaro Kuroiwa, 2015/03/18 01:33:30]

Done.

+    return false;
+  }
+
+  return true;
+}
+
+void TouchInjectorWin::Deinitialize() {
+  touches_in_contact_.clear();
+}
+
+void TouchInjectorWin::InjectTouchEvent(const TouchEvent& event) {
+  if (!delegate_->Initialized()) {
+    VLOG(3) << "Touch injection functions are not initialized.";
+    return;
+  }
+
+  switch (event.event_type()) {
+    case TouchEvent::TOUCH_POINT_START:
+      AddNewTouchPoints(event);
+      break;
+    case TouchEvent::TOUCH_POINT_MOVE:
+      MoveTouchPoints(event);
+      break;
+    case TouchEvent::TOUCH_POINT_END:
+      EndTouchPoints(event);
+      break;
+    case TouchEvent::TOUCH_POINT_CANCEL:
+      CancelTouchPoints(event);
+      break;
+    default:
+      NOTREACHED();
+      return;
+  }
+}
+
+void TouchInjectorWin::SetInjectFunctionsForTest(
+    scoped_ptr<TouchInjectFunctionsWin> functions) {
+  delegate_ = functions.Pass();
+}
+
+void TouchInjectorWin::AddNewTouchPoints(const TouchEvent& event) {
+  DCHECK_EQ(event.event_type(), TouchEvent::TOUCH_POINT_START);
+
+  std::vector<POINTER_TOUCH_INFO> touches;
+  // Must inject already touching points as move events.
+  ChangeToMovePointsAndAppendToVector(&touches_in_contact_, &touches);
+
+  for (const TouchEventPoint& touch_point : event.touch_points()) {
+    POINTER_TOUCH_INFO pointer_touch_info;
+    memset(&pointer_touch_info, 0, sizeof(pointer_touch_info));
+    pointer_touch_info.pointerInfo.pointerFlags =
+        POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT | POINTER_FLAG_DOWN;
+    ConvertToPointerTouchInfo(touch_point, &pointer_touch_info);
+    touches_in_contact_[touch_point.id()] = pointer_touch_info;
+    touches.push_back(pointer_touch_info);
+  }
+
+  if (delegate_->InjectTouchInput(touches.size(),
+                                  vector_as_array(&touches)) == 0) {
+    PLOG(ERROR) << "Failed to inject a touch start event.";
+  }
+}
+
+void TouchInjectorWin::MoveTouchPoints(const TouchEvent& event) {
+  DCHECK_EQ(event.event_type(), TouchEvent::TOUCH_POINT_MOVE);
+
+  for (const TouchEventPoint& touch_point : event.touch_points()) {
+    POINTER_TOUCH_INFO* pointer_touch_info =
+        &touches_in_contact_[touch_point.id()];
+    pointer_touch_info->pointerInfo.pointerFlags =
+        POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT | POINTER_FLAG_UPDATE;
+    ConvertToPointerTouchInfo(touch_point, pointer_touch_info);
+  }
+
+  std::vector<POINTER_TOUCH_INFO> touches;
+  // Must inject already touching points as move events.
+  ChangeToMovePointsAndAppendToVector(&touches_in_contact_, &touches);
+  if (delegate_->InjectTouchInput(touches.size(),
+                                  vector_as_array(&touches)) == 0) {
+    PLOG(ERROR) << "Failed to inject a touch move event.";
+  }
+}
+
+void TouchInjectorWin::EndTouchPoints(const TouchEvent& event) {
+  DCHECK_EQ(event.event_type(), TouchEvent::TOUCH_POINT_END);
+
+  std::vector<POINTER_TOUCH_INFO> touches;
+  for (const TouchEventPoint& touch_point : event.touch_points()) {
+    POINTER_TOUCH_INFO pointer_touch_info =
+        touches_in_contact_[touch_point.id()];
+    pointer_touch_info.pointerInfo.pointerFlags = POINTER_FLAG_UP;
+
+    touches_in_contact_.erase(touch_point.id());
+    touches.push_back(pointer_touch_info);
+  }
+
+  ChangeToMovePointsAndAppendToVector(&touches_in_contact_, &touches);
+  if (delegate_->InjectTouchInput(touches.size(),
+                                  vector_as_array(&touches)) == 0) {
+    PLOG(ERROR) << "Failed to inject a touch end event.";
+  }
+}
+
+void TouchInjectorWin::CancelTouchPoints(const TouchEvent& event) {
+  DCHECK_EQ(event.event_type(), TouchEvent::TOUCH_POINT_CANCEL);
+
+  std::vector<POINTER_TOUCH_INFO> touches;
+  for (const TouchEventPoint& touch_point : event.touch_points()) {
+    POINTER_TOUCH_INFO pointer_touch_info =
+        touches_in_contact_[touch_point.id()];
+    pointer_touch_info.pointerInfo.pointerFlags =
+        POINTER_FLAG_UP | POINTER_FLAG_CANCELED;
+
+    touches_in_contact_.erase(touch_point.id());
+    touches.push_back(pointer_touch_info);
+  }
+
+  ChangeToMovePointsAndAppendToVector(&touches_in_contact_, &touches);
+  if (delegate_->InjectTouchInput(touches.size(),
+                                  vector_as_array(&touches)) == 0) {
+    PLOG(ERROR) << "Failed to inject a touch cancel event.";
+  }
+}
+
+}  // namespace remoting
+