Use devtools to set user agent in chromedriver

Use devtools to set user agent in chromedriver

Use Network.setUserAgentOverride instead of using --user-agent to override the
user agent for mobileEmulation. This is needed to support mobileEmulation with
headless chrome, without adding the --user-agent flag to headless.

BUG=604324

[irisu, 2017-03-30 07:06:17]

irisu@chromium.org changed reviewers:
+ skyostil@chromium.org

[Sami, 2017-03-30 11:21:43]

Thanks, this is what I had in mind. Non-owner lgtm with a couple of comments.

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
File chrome/test/chromedriver/chrome/devtools_http_client.h (right):

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
chrome/test/chromedriver/chrome/devtools_http_client.h:99: const std::string
user_agent();
nit: const std::string& user_agent() const;

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
File chrome/test/chromedriver/chrome/mobile_emulation_override_manager.cc
(right):

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
chrome/test/chromedriver/chrome/mobile_emulation_override_manager.cc:18:
overriden_user_agent_(user_agent) {
typo: overridden

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
chrome/test/chromedriver/chrome/mobile_emulation_override_manager.cc:49: Status
status = client_->SendCommand("Network.enable", empty_params);
Network.enable by default turns on resource caching which can increase memory
use. To avoid that pass maxTotalBufferSize = 0 and maxResourceBufferSize = 0 to
this call.

I assume we don't need to disable the domain anywhere?

[irisu, 2017-03-31 00:10:53]

The CQ bit was checked by irisu@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-31 00:11:42]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-31 00:21:28]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-31 00:21:29]

Dry run: Try jobs failed on following builders:
  linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_tsan_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[irisu, 2017-03-31 00:29:16]

Patchset #2 (id:20001) has been deleted

[irisu, 2017-03-31 00:30:31]

The CQ bit was checked by irisu@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-31 00:31:32]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-31 00:46:21]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-31 00:46:23]

Dry run: Try jobs failed on following builders:
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[irisu, 2017-03-31 01:35:28]

The CQ bit was checked by irisu@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-31 01:36:28]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-31 01:52:33]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-31 01:52:34]

Dry run: This issue passed the CQ dry run.

[irisu, 2017-03-31 01:57:40]

Patchset #2 (id:40001) has been deleted

[irisu, 2017-03-31 01:58:10]

irisu@chromium.org changed reviewers:
+ samuong@chromium.org

[irisu, 2017-03-31 01:58:10]

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
File chrome/test/chromedriver/chrome/devtools_http_client.h (right):

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
chrome/test/chromedriver/chrome/devtools_http_client.h:99: const std::string
user_agent();
On 2017/03/30 11:21:42, Sami wrote:
> nit: const std::string& user_agent() const;

Done.

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
File chrome/test/chromedriver/chrome/mobile_emulation_override_manager.cc
(right):

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
chrome/test/chromedriver/chrome/mobile_emulation_override_manager.cc:18:
overriden_user_agent_(user_agent) {
On 2017/03/30 11:21:42, Sami wrote:
> typo: overridden

Done.

https://codereview.chromium.org/2785083002/diff/1/chrome/test/chromedriver/ch...
chrome/test/chromedriver/chrome/mobile_emulation_override_manager.cc:49: Status
status = client_->SendCommand("Network.enable", empty_params);
On 2017/03/30 11:21:42, Sami wrote:
> Network.enable by default turns on resource caching which can increase memory
> use. To avoid that pass maxTotalBufferSize = 0 and maxResourceBufferSize = 0
to
> this call.
> 
> I assume we don't need to disable the domain anywhere?

Done.

I don't believe it needs to be disabled - it's also enabled for network
emulation.

[irisu, 2017-04-03 03:31:43]

irisu@chromium.org changed reviewers:
+ stgao@chromium.org
- samuong@chromium.org

[johnchen, 2017-04-03 14:58:39]

lgtm

[stgao, 2017-04-03 18:28:52]

This looks good to me in general. And johnchen@'s approval is sufficient now for
CQ as he is the new owner and has just been added to the OWNERS file.

However, I have a question on this scenario:
Now we connect to the current TAB and instruct DevTools to simulate a given
user-agent. But clicking on a link in the current TAB opens a new TAB for
another page from the same server host. In the new TAB, will DevTools
automatically simulate the same user-agent? What if it is from a different
server host?

cced srawlins@ who implemented this feature a long while ago. IIRC, there was a
design doc on this.

[irisu, 2017-04-05 03:08:48]

The CQ bit was checked by irisu@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-05 03:09:06]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[irisu, 2017-04-05 03:10:12]

On 2017/04/03 18:28:52, stgao(ping after 24h) wrote:
> This looks good to me in general. And johnchen@'s approval is sufficient now
for
> CQ as he is the new owner and has just been added to the OWNERS file.
> 
> However, I have a question on this scenario:
> Now we connect to the current TAB and instruct DevTools to simulate a given
> user-agent. But clicking on a link in the current TAB opens a new TAB for
> another page from the same server host. In the new TAB, will DevTools
> automatically simulate the same user-agent? What if it is from a different
> server host?
> 
> cced srawlins@ who implemented this feature a long while ago. IIRC, there was
a
> design doc on this.

PTAL. Added an extra test for the user agent, after having clicking on a link
which
opens in a new tab.

[commit-bot: I haz the power, 2017-04-05 03:21:16]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-05 03:21:17]

Dry run: This issue passed the CQ dry run.

[stgao, 2017-04-05 04:17:08]

Sorry, I might have confused you on the user-agent of the new-tab thing.

I commented on the new test. Hope I made it more clear this time.

https://codereview.chromium.org/2785083002/diff/80001/chrome/test/chromedrive...
File chrome/test/chromedriver/test/run_py_tests.py (right):

https://codereview.chromium.org/2785083002/diff/80001/chrome/test/chromedrive...
chrome/test/chromedriver/test/run_py_tests.py:1915: driver.MouseClick()
What's the user-agent if the new tab itself directly loads the "/userAgent"
below before Devtools connects to the new tab and instructs it to simulate the
user-agent?

https://codereview.chromium.org/2785083002/diff/80001/chrome/test/chromedrive...
chrome/test/chromedriver/test/run_py_tests.py:1918:
driver.Load(self._http_server.GetUrl() + '/userAgent')
Before this load, we already instruct the new tab to simulate the user-agent
above, so this load will return the simulated one for sure.

I'm curious about the user-agent of the page that the new tab loads by itself
when opened.

[irisu, 2017-04-05 06:44:33]

https://codereview.chromium.org/2785083002/diff/80001/chrome/test/chromedrive...
File chrome/test/chromedriver/test/run_py_tests.py (right):

https://codereview.chromium.org/2785083002/diff/80001/chrome/test/chromedrive...
chrome/test/chromedriver/test/run_py_tests.py:1915: driver.MouseClick()
On 2017/04/05 04:17:08, stgao(ping after 24h) wrote:
> What's the user-agent if the new tab itself directly loads the "/userAgent"
> below before Devtools connects to the new tab and instructs it to simulate the
> user-agent?

Sorry for that misunderstanding. Modified the test and you are correct - this
test now fails.

I will try looking at the docs for a fix, otherwise it may be best to add the
user-agent flag to headless after all to avoid this.

[Sami, 2017-04-05 13:52:04]

https://codereview.chromium.org/2785083002/diff/80001/chrome/test/chromedrive...
File chrome/test/chromedriver/test/run_py_tests.py (right):

https://codereview.chromium.org/2785083002/diff/80001/chrome/test/chromedrive...
chrome/test/chromedriver/test/run_py_tests.py:1915: driver.MouseClick()
On 2017/04/05 06:44:33, irisu wrote:
> On 2017/04/05 04:17:08, stgao(ping after 24h) wrote:
> > What's the user-agent if the new tab itself directly loads the "/userAgent"
> > below before Devtools connects to the new tab and instructs it to simulate
the
> > user-agent?
> 
> Sorry for that misunderstanding. Modified the test and you are correct - this
> test now fails.
> 
> I will try looking at the docs for a fix, otherwise it may be best to add the
> user-agent flag to headless after all to avoid this.

Right, looks like we need to support a default user agent after all for this
reason.

[irisu, 2017-04-07 06:05:31]

Description was changed from

==========
Use devtools to set user agent in chromedriver

Use Network.setUserAgentOverride instead of using --user-agent to override the
user agent for mobileEmulation. This is needed to support mobileEmulation with
headless chrome, without adding the --user-agent flag to headless.

BUG=604324
==========

to

==========
Use devtools to set user agent in chromedriver

Use Network.setUserAgentOverride instead of using --user-agent to override the
user agent for mobileEmulation. This is needed to support mobileEmulation with
headless chrome, without adding the --user-agent flag to headless.

BUG=604324
==========