Chromium Code Reviews
|
|
Chromium Code Reviews
Issue
1680143002:
Simulate click through gamepad BUTTON_A (Closed)
|
Created:
4 years, 10 months ago by sshelke Modified:
3 years, 9 months ago CC:
blink-reviews, blink-reviews-html_chromium.org, chromium-reviews, dglazkov+blink, dmazzoni, eric.carlson_apple.com, feature-media-reviews_chromium.org, fs, gasubic, mlamouri+watch-blink_chromium.org, philipj_slow, nessy, tdresser+watch_chromium.org, vcarbune.chromium Base URL:
https://chromium.googlesource.com/chromium/src.git@master Target Ref:
refs/pending/heads/master Project:
chromium Visibility:
Public. |
DescriptionSimulate click through gamepad BUTTON_A
This change adds logic to simulate click on
focused html element through KEYCODE_BUTTON_A event.
BUG=454355
Patch Set 1 #
Total comments: 5
Patch Set 2 : simulate click #
Messages
Total messages: 47 (11 generated)
sshelke@nvidia.com changed reviewers: + aelias@chromium.org, rbyers@chromium.org, tdresser@chromium.org, wez@chromium.org
Uploaded initial patch. Please review.
dtapuska@chromium.org changed reviewers: + dtapuska@chromium.org
https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... File third_party/WebKit/Source/core/html/HTMLMediaElement.cpp (right): https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... third_party/WebKit/Source/core/html/HTMLMediaElement.cpp:3493: if (event->type() == EventTypeNames::keydown && toKeyboardEvent(event)->keyIdentifier() == "Enter") { I'd prefer if this was using domCode() as opposed to keyIdentifier() if you want to use a Stringized version. Likewise this can be written with an integer; in terms of ->keyCode() == '\r' https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... File third_party/WebKit/Source/core/html/forms/BaseCheckableInputType.cpp (left): https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... third_party/WebKit/Source/core/html/forms/BaseCheckableInputType.cpp:62: const String& key = event->keyIdentifier(); As per the similar comment before; keyIdentifier will eventually be removed. So using domCode() or keyCode() is preferred for now. Currently we don't have domKey() working everywhere so ideally you'd use that. Personally I'd use the keyCode() == ' ' || keyCode() == '\r'
https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... File third_party/WebKit/Source/core/html/HTMLMediaElement.cpp (right): https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... third_party/WebKit/Source/core/html/HTMLMediaElement.cpp:3493: if (event->type() == EventTypeNames::keydown && toKeyboardEvent(event)->keyIdentifier() == "Enter") { On 2016/02/09 14:29:03, dtapuska wrote: > I'd prefer if this was using domCode() as opposed to keyIdentifier() if you want > to use a Stringized version. > > Likewise this can be written with an integer; in terms of ->keyCode() == '\r' > Seeing what you are doing later on; mapping a the BUTTON_A to enter; I recommend using the keyCode.. https://codereview.chromium.org/1680143002/diff/1/ui/events/keycodes/keyboard... File ui/events/keycodes/keyboard_code_conversion_android.cc (right): https://codereview.chromium.org/1680143002/diff/1/ui/events/keycodes/keyboard... ui/events/keycodes/keyboard_code_conversion_android.cc:102: case AKEYCODE_BUTTON_A: There is a comment that you didn't remove at the end of this method that has a // case AKEYCODE_BUTTON_A: It should likely be removed.
This patch doesn't make much sense. First, mapping button-A to enter is totally arbitrary and surprising. Secondly, Enter didn't even have all the behaviors you wanted before, so you're not even tapping into the natural default behavior by doing this. If you want to change checkboxes to react to button-A, that seems reasonable, but let's not fake any keyboard keycodes to do do so. You can introduce a button-A-specific keycode and make checkboxes react to that. https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... File third_party/WebKit/Source/core/html/forms/RadioInputType.cpp (right): https://codereview.chromium.org/1680143002/diff/1/third_party/WebKit/Source/c... third_party/WebKit/Source/core/html/forms/RadioInputType.cpp:130: if (key != "U+0020" && key != "Enter") For some reason, you're changing behavior globally on all platforms to make Enter select radio buttons (and the other stuff) when it was only Spacebar that had that effect before. I don't see any reason why that change should be made.
> If you want to change checkboxes to react to button-A, that seems reasonable, > but let's not fake any keyboard keycodes to do do so. You can introduce a > button-A-specific keycode and make checkboxes react to that. Could you please let me know standard dom key value for button-A? If I know dom key value for button-A as per standard values provided in https://www.w3.org/TR/DOM-Level-3-Events-code/ then I can create new button-A specific keycode. If there is need to create custom keycode for button-A, do you suggest to include new non-standard keycode in https://code.google.com/p/chromium/codesearch#chromium/src/ui/events/keycodes... which contains all standard dom keycode values?
Maybe even better would be not to introduce a fake keyboard keycode at all but make the relevant elements react directly to gamepad events.
On 2016/02/10 20:23:15, aelias wrote: > Maybe even better would be not to introduce a fake keyboard keycode at all but > make the relevant elements react directly to gamepad events. That's my main concern, in order to make html elements react to gamepad events, we should have standard dom values associated with it, unfortunately W3C specification don't provide standard dom values for gamepad events. We might have to use non-standard dom values for gamepad events. Will that be okay?
Description was changed from ========== Emulate gamepad BUTTON_A as enter key This change adds logic to emulate gamepad BUTTON_A as "Enter" key to achieve following functionalities. 1) Toggle html media element play state. 2) Click on html link. 3) HTML form - change checkbox and radio button input state. BUG=454355 ========== to ========== Emulate gamepad BUTTON_A as enter key This change adds logic to emulate gamepad BUTTON_A as "Enter" key to achieve following functionalities. 1) Toggle html media element play state. 2) Click on html link. 3) HTML form - change checkbox and radio button input state. BUG=454355 ==========
dtapuska@chromium.org changed reviewers: + garykac@chromium.org
On 2016/02/11 06:24:03, sshelke wrote: > On 2016/02/10 20:23:15, aelias wrote: > > Maybe even better would be not to introduce a fake keyboard keycode at all but > > make the relevant elements react directly to gamepad events. > > That's my main concern, in order to make html elements react to gamepad events, > we should have standard dom values associated with it, unfortunately W3C > specification don't provide standard dom values for gamepad events. > We might have to use non-standard dom values for gamepad events. > Will that be okay? +Gary is this something that we will add to the spec?
garykac@chromium.org changed reviewers: + scottmg@chromium.org
On 2016/02/16 17:58:05, dtapuska wrote: > On 2016/02/11 06:24:03, sshelke wrote: > > On 2016/02/10 20:23:15, aelias wrote: > > > Maybe even better would be not to introduce a fake keyboard keycode at all > but > > > make the relevant elements react directly to gamepad events. > > > > That's my main concern, in order to make html elements react to gamepad > events, > > we should have standard dom values associated with it, unfortunately W3C > > specification don't provide standard dom values for gamepad events. > > We might have to use non-standard dom values for gamepad events. > > Will that be okay? > > +Gary is this something that we will add to the spec? +scottmg (for Gamepad spec) Do the gamepad buttons generate keydown/keyup events? I don't see that mentioned in the spec. My reading is that people should be using getGamepads() to read the button state. If we expect the gamepad to generate keyboard events, then: (1) That should be spec'ed somewhere (probably in the gamepad spec) (2) We should add a location (DOM_KEY_LOCATION_GAMEPAD) (in UIEvents spec) (3) We should specify the |code| and |key| values that should be used for the standard gamepad keys (in UIEvents) We removed some old info from the UIEvents spec (like DOM_KEY_LOCATION_JOYSTICK) because our assumption was that the Gamepad api would be how people would query the gamepad state. If we want gamepad buttons to act like keyboard buttons, then we need to spec this and get consensus from other browser vendors before going too far down this road.
On 2016/02/17 22:15:00, garykac wrote: > On 2016/02/16 17:58:05, dtapuska wrote: > > On 2016/02/11 06:24:03, sshelke wrote: > > > On 2016/02/10 20:23:15, aelias wrote: > > > > Maybe even better would be not to introduce a fake keyboard keycode at all > > but > > > > make the relevant elements react directly to gamepad events. > > > > > > That's my main concern, in order to make html elements react to gamepad > > events, > > > we should have standard dom values associated with it, unfortunately W3C > > > specification don't provide standard dom values for gamepad events. > > > We might have to use non-standard dom values for gamepad events. > > > Will that be okay? > > > > +Gary is this something that we will add to the spec? > > +scottmg (for Gamepad spec) > > Do the gamepad buttons generate keydown/keyup events? I don't see that mentioned > in the spec. My reading is that people should be using getGamepads() to read the > button state. > > If we expect the gamepad to generate keyboard events, then: > (1) That should be spec'ed somewhere (probably in the gamepad spec) > (2) We should add a location (DOM_KEY_LOCATION_GAMEPAD) (in UIEvents spec) > (3) We should specify the |code| and |key| values that should be used for the > standard gamepad keys (in UIEvents) > > We removed some old info from the UIEvents spec (like DOM_KEY_LOCATION_JOYSTICK) > because our assumption was that the Gamepad api would be how people would query > the gamepad state. > > If we want gamepad buttons to act like keyboard buttons, then we need to spec > this and get consensus from other browser vendors before going too far down this > road. No, there's no provision for making gamepads be poor keyboards.
>> Do the gamepad buttons generate keydown/keyup events? Yes, gamepad buttons generate keydown/keyup events. We just don't know dom value for gamepad buttons which can be passed to blink. >> I don't see that mentioned in the spec. I guess, previously there were no specific functionalities assigned to gamepad events other than using those events in gamepad API. That's why gampead is not part of spec. But now we have requirements mentioned in https://docs.google.com/document/d/1rUHKwLCCwJKnnVAFnFAm13uBAc2u2YfJGF36LdUSv... >> My reading is that people should be using getGamepads() to read the >> button state. getGamepads are used for accessing gameapd through Gamepad API. But here requirement is quite different, we wish to operate BUTTON_A as enter when gamepad API is not active in current tab. When browser is opened with --enable-spatial-navigation flag and user navigate between HTML elements using D-PAD buttons. Currently, there is no provision for user to click on html link using gamepad buttons. Generally users are inclined towards using BUTTON_A as enter.We have already assigned some functionalty to gamepad buttons such as BUTTON_X = focus url bar BUTTON_Y = bring up menu BUTTON_B = Close tab Only BUTTON_A is left for using it as means to behave as "Enter". Can't we just emulate BUTTON_A as some other key instead of making gamepad buttons as part of specs?
I agree we really don't want to be changing the behavior on other platforms
On 2016/02/18 22:25:25, Rick Byers (out until Feb 18) wrote: > I agree we really don't want to be changing the behavior on other platforms Hah, whoops sorry - tested what hitting enter on a checkbox did (apparently it submits the form!). So yeah, we can't just go changing this behavior without more careful consideration. So the problem here is that you want to trigger both link navigation and checkbox selection with the same event, and no such event exists today. What about links that rely on an 'onclick' handler, presumably you want to trigger those too - right? What about text fields, should hitting BUTTON_A really insert a new line? Or what about JavaScript that's listenting to keyboard events. Figuring out how best to expose this behavior to the web seems the trickiest part to me. However, that problem seems very similar to me to accessibility scenarios, eg. TalkBack on Android. I'd suggest investigating how activation (link navigation + checkbox toggling etc.) works in Chrome with TalkBack. Perhaps that'll give us a nice generic way to have the chromium UI layer say "activate the focused thing now" without any sort of simulating different input devices (such simulation often comes back to haunt us - eg. the history of "click" is very complex and sad).
rbyers@chromium.org changed reviewers: - rbyers@chromium.org, tdresser@chromium.org
On 2016/02/18 22:39:18, Rick Byers (out until Feb 18) wrote: > On 2016/02/18 22:25:25, Rick Byers (out until Feb 18) wrote: > > I agree we really don't want to be changing the behavior on other platforms > > Hah, whoops sorry - tested what hitting enter on a checkbox did (apparently it > submits the form!). So yeah, we can't just go changing this behavior without > more careful consideration. > > So the problem here is that you want to trigger both link navigation and > checkbox selection with the same event, and no such event exists today. What > about links that rely on an 'onclick' handler, presumably you want to trigger > those too - right? What about text fields, should hitting BUTTON_A really > insert a new line? Or what about JavaScript that's listenting to keyboard > events. Figuring out how best to expose this behavior to the web seems the > trickiest part to me. > > However, that problem seems very similar to me to accessibility scenarios, eg. > TalkBack on Android. I'd suggest investigating how activation (link navigation > + checkbox toggling etc.) works in Chrome with TalkBack. Perhaps that'll give > us a nice generic way to have the chromium UI layer say "activate the focused > thing now" without any sort of simulating different input devices (such > simulation often comes back to haunt us - eg. the history of "click" is very > complex and sad). /cc dmazzoni@chromium.org - accessibility expert.
rbyers@chromium.org changed reviewers: + rbyers@chromium.org
Can we emulate Enter for BUTTON_A for clicking links only? Clicking link is major requirement. Currently when user navigate through html links using gamepad D-Pad keys, there is no mapping available for gamepad buttons to click link. Currently,enter key is used for clicking links on all platforms so we wont be breaking any functionality. Other requirements such as filling forms, media element toggle etc, can be fulfilled through other means.
On 2016/03/01 06:22:12, sshelke wrote: > Can we emulate Enter for BUTTON_A for clicking links only? > Clicking link is major requirement. Currently when user navigate > through html links using gamepad D-Pad keys, there is no mapping > available for gamepad buttons to click link. > Currently,enter key is used for clicking links on all platforms > so we wont be breaking any functionality. Yes, I think that's basically what we've been talking about. But only blink knows the difference between links and other elements, so this probably isn't really "emulating enter" anymore. You need to send some sort of event to blink which, when it occurs on a link, is handled the same as the enter key but is otherwise ignored (or possibly handled differently on different elements). This is why I suggested looking into how TalkBack / accessibility works - it needs to do something very much like that so there may be some existing infrastructure you can re-use. An alternative may be to create a new type of WebInputEvent (eg. 'WebGamepadEvent') that is similar to but distinct from WebKeyboardEvent. Or maybe it's not terrible to have some code higher in the stack ask blink what type of node currently has focus, and if you find it's a link then send a fake enter keypress event. Ultimately the key difference in a lot of these designs is exactly how the action is exposed to JavaScript (eg. are there keydown and keyup events, and if so what are their exact properties?). This is an area that accessibility folks have struggled with for years - balancing web compatibility with a rational programming model for web developers when unusual input mechanisms are being used. So again I expect there's a lot to learn from how this is handled by TalkBack. dmazzoni@ any advice? > > Other requirements such as filling forms, media element > toggle etc, can be fulfilled through other means.
dmazzoni@chromium.org changed reviewers: + dmazzoni@chromium.org
Rick asked about Android, but that's actually an unusual case. Let's talk about the usual cases first. First, users who rely on a keyboard need to be able to use space to toggle a checkbox and enter to submit a form. Please don't break that! Next, on most platforms there's a way to "activate" an element via accessibility APIs. For most elements that results in calling AXObject::press(), which puts a UserGestureIndicator in scope and then calls accessKeyAction. Most elements implement this by focusing and then simulating a click. That's what happens if a blind user presses their screen reader's special keystroke to activate/click the current item, for example - it calls that, it doesn't send Space or Enter. Would it make sense to do the same thing for BUTTON_A? Either call accessKeyAction, since that's more or less what it's designed for, or else just use UserGestureIndicator and simulate a click directly? TalkBack on Android is a bit different. For various historical reasons, when you activate the current element in TalkBack it simulates touch events in the center of the element with accessibility focus. Partly this is so things like dragging and long-pressing can work. However, users who use Android via a braille display end up triggering AXObject::press() just like on other platforms. Hope that helps.
On 2016/03/01 17:46:56, dmazzoni wrote: > Rick asked about Android, but that's actually an unusual case. Let's talk about > the usual cases first. > > First, users who rely on a keyboard need to be able to use space to toggle a > checkbox and enter to submit a form. Please don't break that! > > Next, on most platforms there's a way to "activate" an element via accessibility > APIs. For most elements that results in calling AXObject::press(), which puts a > UserGestureIndicator in scope and then calls accessKeyAction. Most elements > implement this by focusing and then simulating a click. That's what happens if a > blind user presses their screen reader's special keystroke to activate/click the > current item, for example - it calls that, it doesn't send Space or Enter. > > Would it make sense to do the same thing for BUTTON_A? Either call > accessKeyAction, since that's more or less what it's designed for, or else just > use UserGestureIndicator and simulate a click directly? > > TalkBack on Android is a bit different. For various historical reasons, when you > activate the current element in TalkBack it simulates touch events in the center > of the element with accessibility focus. Partly this is so things like dragging > and long-pressing can work. However, users who use Android via a braille display > end up triggering AXObject::press() just like on other platforms. > > Hope that helps. Thanks Dominic, that's helpful! Is there already an easy way to get the WebAXObject for the currently focused element? I don't think WebView exposes the currently focused element, does it? If not, one option is to add a new API to WebView similar to showContextMenu that's implemented by getting the AXObject for the currently focused element and invoking "press" on it.
On Tue, Mar 1, 2016 at 12:00 PM <rbyers@chromium.org> wrote: > Thanks Dominic, that's helpful! Is there already an easy way to get the > WebAXObject for the currently focused element? I don't think WebView > exposes > the currently focused element, does it? > You want to do this from Chromium using public blink APIs? You could do it via accessibility using only public Blink APIs - first create a WebScopedAXContext, then call focusedAccessibilityObject() on the focused WebDocument, then call performDefaultAction() on that. We haven't really optimized for that case, though, so I'd prefer if you didn't use WebScopedAXContext for this. Actually the first thing I'd try is calling WebNode::simulateClick(). It's possible that already works. If that doesn't create a UserGestureIndicator, though, you might want to modify it to take a bool parameter for that, or create a parallel API that creates a UserGestureIndicator and calls accessKeyAction like AXObject does. - Dominic -- You received this message because you are subscribed to the Google Groups "Blink Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-reviews+unsubscribe@chromium.org.
On Tue, Mar 1, 2016 at 12:00 PM <rbyers@chromium.org> wrote: > Thanks Dominic, that's helpful! Is there already an easy way to get the > WebAXObject for the currently focused element? I don't think WebView > exposes > the currently focused element, does it? > You want to do this from Chromium using public blink APIs? You could do it via accessibility using only public Blink APIs - first create a WebScopedAXContext, then call focusedAccessibilityObject() on the focused WebDocument, then call performDefaultAction() on that. We haven't really optimized for that case, though, so I'd prefer if you didn't use WebScopedAXContext for this. Actually the first thing I'd try is calling WebNode::simulateClick(). It's possible that already works. If that doesn't create a UserGestureIndicator, though, you might want to modify it to take a bool parameter for that, or create a parallel API that creates a UserGestureIndicator and calls accessKeyAction like AXObject does. - Dominic -- You received this message because you are subscribed to the Google Groups "Chromium-reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to chromium-reviews+unsubscribe@chromium.org.
Is there an update on this? As I understand it:
(1) Currently, Chromium sends keydown/keyup events for gamepad key
presses (according to sshelke)
(2) Neither the Gamepad nor UIEvents spec curently describes these
key events
(3) Gamepad spec does not want to add a "provision for making
gamepads be poor keyboards" (inferring from scottmg's comment)
It sounds like we need to either:
(a) Spec what gamepad-generated key events look like, or
(b) File a bug to have these events no longer fire in Chromium for
gamepads.
I don't know what other browsers do, but if they all fire these events,
then we probably need to do (a).
Continuing the list...
(4) According to the Gamepad Interaction spec[1], BUTTON_A "should be
treated as Enter". In particular, BUTTON_A should follow links
when they are selected.
(5) But "treated like Enter" is not strictly true, since the desired
behavior includes selecting checkboxes, which Enter does not
currently do (nor is that a desired change).
Here is sounds like we need either:
(c) Use keydown/keyup and define default behaviors for these events.
(d) Hook deeper into the input stack and simulate a click (as dmazzoni
describes) when BUTTON_A is clicked on links and checkboxes
If (a), then (c). If (b), then (d).
It's not hard to spec what these keydown/keyup events would look like,
and there's the benefit of having the dpad acting like arrow keys.
If we want to go that route (specing keydown/up for gamepad), I would
propose updating UIEvents to include something like:
|code| |key|
BUTTON_A GamepadA GamepadA
BUTTON_B GamepadB GamepadB
BUTTON_X GamepadX GamepadX
BUTTON_Y GamepadY GamepadY
BUTTON_L1 GamepadLeft1 GamepadLeft1
BUTTON_L2 GamepadLeft2 GamepadLeft2
BUTTON_R1 GamepadRight1 GamepadRight1
BUTTON_R2 GamepadRight2 GamepadRight2
DPAD_DOWN GamepadArrowDown ArrowDown
DPAD_LEFT GamepadArrowLeft ArrowLeft
DPAD_RIGHT GamepadArrowRight ArrowRight
DPAD_UP GamepadArrowUp ArrowUp
START GamepadStart MediaPlay (?)
VOLUME GamepadVolume AudioVolumeMute (?)
BACK GamepadBack GoBack
HOME GamepadHome GoHome
KeyboardEvent.location would be set to DOM_KEY_LOCATION_GAMEPAD for
all these keys.
[1]
https://docs.google.com/document/d/1rUHKwLCCwJKnnVAFnFAm13uBAc2u2YfJGF36LdUSv...
Thanks garykac for brief explanation. It seems "If (a), then (c)" seems cleaner way but Currently I am trying " If (b), then (d)" using approaches provided by dmazzoni and Rick i.e clicking link through accessibility object or simulate link after BUTTON_A is pressed. We have intercepted buttons BUTTON_B/X/Y/L1/L2, START, HOME at UI level for handling UI operations on android. Please refer https://codereview.chromium.org/1276703003. I needed dom key for BUTTON_A to achieve following operations. 1) Toggle media playback 2) Click on check boxes, radio buttons (i.e 3) Click on link If we get dom key for BUTTON_A i.e GamepadA, above operations can be easily achieved.
On 2016/03/18 06:56:12, sshelke wrote: > Thanks garykac for brief explanation. It seems "If (a), then (c)" seems > cleaner way but Currently I am trying " If (b), then (d)" using approaches > provided by dmazzoni and Rick i.e clicking link through accessibility object > or simulate link after BUTTON_A is pressed. > > We have intercepted buttons BUTTON_B/X/Y/L1/L2, START, HOME at UI level > for handling UI operations on android. Please refer > https://codereview.chromium.org/1276703003. > I needed dom key for BUTTON_A to achieve following operations. > 1) Toggle media playback > 2) Click on check boxes, radio buttons (i.e > 3) Click on link > > If we get dom key for BUTTON_A i.e GamepadA, above operations can > be easily achieved. I was suggesting the b/d path over a/c mainly because I didn't think an implementation detail like this should necessarily influence web API design (and didn't want it to be blocked on unpredictable spec consensus etc.). However if you (Gary) think it's reasonable (and likely not too controvercial) to extend the spec to have keyboard events for gamepad buttons then maybe that is a better approach? That's probably best discussed in the UIEvents WG though - I've filed https://github.com/w3c/uievents/issues/79, how about we continue the discussion there? sshelke@ feel free to proceed on the b/d path if it's looking reasonable to you. But it's possible we could get consensus on this spec issue relatively quickly, in which case you could switch to the a/c approach if you prefer (though that will require an intent-to-ship per the blink API launch policy).
On 2016/03/18 15:52:42, Rick Byers wrote: > On 2016/03/18 06:56:12, sshelke wrote: > > Thanks garykac for brief explanation. It seems "If (a), then (c)" seems > > cleaner way but Currently I am trying " If (b), then (d)" using approaches > > provided by dmazzoni and Rick i.e clicking link through accessibility object > > or simulate link after BUTTON_A is pressed. > > > > We have intercepted buttons BUTTON_B/X/Y/L1/L2, START, HOME at UI level > > for handling UI operations on android. Please refer > > https://codereview.chromium.org/1276703003. > > I needed dom key for BUTTON_A to achieve following operations. > > 1) Toggle media playback > > 2) Click on check boxes, radio buttons (i.e > > 3) Click on link > > > > If we get dom key for BUTTON_A i.e GamepadA, above operations can > > be easily achieved. > > I was suggesting the b/d path over a/c mainly because I didn't think an > implementation detail like this should necessarily influence web API design (and > didn't want it to be blocked on unpredictable spec consensus etc.). > > However if you (Gary) think it's reasonable (and likely not too controvercial) > to extend the spec to have keyboard events for gamepad buttons then maybe that > is a better approach? That's probably best discussed in the UIEvents WG though > - I've filed https://github.com/w3c/uievents/issues/79, how about we continue > the discussion there? > > sshelke@ feel free to proceed on the b/d path if it's looking reasonable to you. > But it's possible we could get consensus on this spec issue relatively quickly, > in which case you could switch to the a/c approach if you prefer (though that > will require an intent-to-ship per the blink API launch policy). I investigated this further on Safari/FF/Chrome on Mac and none of the browsers generated keyboard events for the gamepad. Given this, a/c is not really an option (since we don't want to *add* that functionality - we only wanted to document existing/shipping functionality). So b/d makes the most sense.
On 2016/04/05 23:57:09, garykac wrote: > On 2016/03/18 15:52:42, Rick Byers wrote: > > On 2016/03/18 06:56:12, sshelke wrote: > > > Thanks garykac for brief explanation. It seems "If (a), then (c)" seems > > > cleaner way but Currently I am trying " If (b), then (d)" using approaches > > > provided by dmazzoni and Rick i.e clicking link through accessibility object > > > > or simulate link after BUTTON_A is pressed. > > > > > > We have intercepted buttons BUTTON_B/X/Y/L1/L2, START, HOME at UI level > > > for handling UI operations on android. Please refer > > > https://codereview.chromium.org/1276703003. > > > I needed dom key for BUTTON_A to achieve following operations. > > > 1) Toggle media playback > > > 2) Click on check boxes, radio buttons (i.e > > > 3) Click on link > > > > > > If we get dom key for BUTTON_A i.e GamepadA, above operations can > > > be easily achieved. > > > > I was suggesting the b/d path over a/c mainly because I didn't think an > > implementation detail like this should necessarily influence web API design > (and > > didn't want it to be blocked on unpredictable spec consensus etc.). > > > > However if you (Gary) think it's reasonable (and likely not too controvercial) > > to extend the spec to have keyboard events for gamepad buttons then maybe that > > is a better approach? That's probably best discussed in the UIEvents WG > though > > - I've filed https://github.com/w3c/uievents/issues/79, how about we continue > > the discussion there? > > > > sshelke@ feel free to proceed on the b/d path if it's looking reasonable to > you. > > But it's possible we could get consensus on this spec issue relatively > quickly, > > in which case you could switch to the a/c approach if you prefer (though that > > will require an intent-to-ship per the blink API launch policy). > > I investigated this further on Safari/FF/Chrome on Mac and none of the browsers > generated keyboard events for the gamepad. Given this, a/c is not really an > option (since we don't want to *add* that functionality - we only wanted to > document existing/shipping functionality). > > So b/d makes the most sense. Sounds like we can close this patch, then?
On 2016/04/09 01:01:19, Wez wrote:
> Sounds like we can close this patch, then?
No, the bug still needs to be fixed. It just needs to go the b/d implementation
route, where:
(b) File a bug to have these events no longer fire in Chromium for
gamepads.
(d) Hook deeper into the input stack and simulate a click (as dmazzoni
describes) when BUTTON_A is clicked on links and checkboxes
And (b) is not an issue (since, according to my tests, Chromium does not
currently fire these events), that means that sshelke@ just needs to continue
working on (d) to fix this.
Description was changed from ========== Emulate gamepad BUTTON_A as enter key This change adds logic to emulate gamepad BUTTON_A as "Enter" key to achieve following functionalities. 1) Toggle html media element play state. 2) Click on html link. 3) HTML form - change checkbox and radio button input state. BUG=454355 ========== to ========== Simulate click through gamepad BUTTON_A This change adds logic to simulate click on focused html element through KEYCODE_BUTTON_A event. BUG=454355 ==========
Added patch which simulates click on focused element when gamepad BUTTON_A is pressed. Please review and let me know your feedback.
On 2016/05/09 11:48:10, sshelke wrote: > Added patch which simulates click on focused element when gamepad > BUTTON_A is pressed. Please review and let me know your feedback. PTAL, Thanks.
I don't think we should be adding any extra IPC cruft or Android-specific code for this simulated click concept. AFAIK the renderer process already receives all necessary information about the gamepad buttons.
On 2016/05/11 18:50:52, aelias wrote: > I don't think we should be adding any extra IPC cruft or Android-specific code > for this simulated click concept. AFAIK the renderer process already receives > all necessary information about the gamepad buttons. You mean for the gamepad API? We only send those IPCs when someone is actively using the gamepad API IIRC (and in those cases we probably don't want any gamepad-derived click). Something like this seems pretty reasonable to me. But I'll leave the review to you since this is all above blink now.
rbyers@chromium.org changed reviewers: - dmazzoni@chromium.org
OK. Well, Android framework is actually already generating fake keyboard events for BUTTON_A for us, and by far the most natural way to plumb this down to Blink is to add those key types to the appropriate enums, instead of this weird new backchannel. I also think that this behavior ought to be preventDefault()able and the key event is a natural way to do that. Reading the discussion above, the main argument against this route is that desktop browsers do not have such gamepad key events. I'm not sure that's a compelling argument in this case, since all we'd be doing is reflecting the preexisting platform differences (I think these Android key events even probably already are web-visible, only with some manner of "unknown" key code), as opposed to going out of our way to hide information Android provides to us just because there's no analogy on desktop.
On 2016/05/12 02:36:40, aelias wrote: > OK. Well, Android framework is actually already generating fake keyboard events > for BUTTON_A for us, and by far the most natural way to plumb this down to Blink > is to add those key types to the appropriate enums, instead of this weird new > backchannel. I also think that this behavior ought to be preventDefault()able > and the key event is a natural way to do that. > > Reading the discussion above, the main argument against this route is that > desktop browsers do not have such gamepad key events. I'm not sure that's a > compelling argument in this case, since all we'd be doing is reflecting the > preexisting platform differences (I think these Android key events even probably > already are web-visible, only with some manner of "unknown" key code), as > opposed to going out of our way to hide information Android provides to us just > because there's no analogy on desktop. There's a reasonable argument there I think. But this isn't the forum in which to make it. Alex, want to chime in on https://github.com/w3c/uievents/issues/79 ?
>> AFAIK the renderer process already receives all necessary information about the gamepad buttons. It is true that render_widget receives all keyboard events but those events get translated into WebKeyBoardEvent in keyboard_code_conversion_android. BUTTON_A gets translated to DomKey::NONE in key conversion process. Until and unless we have some mapping like DomKey::BUTTON_A, we cant proceed further with approach (a) and (b) as suggested by garykac. I guess, discussion for approach (a) and (b) is already in progress on https://github.com/w3c/uievents/issues/79. I am trying approach (b) and (d) till any conclusion comes out from https://github.com/w3c/uievents/issues/79. It would be helpful, if you can suggest alternative solution which does not use IPC for (b) and (d) approach.
This feature is nonurgent so I'm OK not landing anything for this until all the spec discussion is sorted out.
On 2016/05/12 20:35:45, aelias wrote: > This feature is nonurgent so I'm OK not landing anything for this until all the > spec discussion is sorted out. @aelias Could you please let me know whether spec discussion is sorted out or not? I think it would take time to sort out some things about specs, meanwhile could you review the patch or suggest some alternative if you are not happy with current the solution. I know this patch is non-urgent or may be not so important but keeping it idle would not do any good either.
wez@chromium.org changed reviewers: - wez@chromium.org
-me |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
