Add variations combinations testing framework

Add variations combinations testing framework

BUG=714827
Review-Url: https://codereview.chromium.org/2843463004
Cr-Commit-Position: refs/heads/master@{#468025}
Committed: https://chromium.googlesource.com/chromium/src/+/fb645f073cb67786bb972c80edcde253f733b29c

[Robert Ogden, 2017-04-24 22:21:58]

Description was changed from

==========
Add variations combinations testing framework

BUG=
==========

to

==========
Add variations combinations testing framework
==========

[Robert Ogden, 2017-04-24 22:23:13]

robertogden@chromium.org changed reviewers:
+ ryansturm@chromium.org

[Robert Ogden, 2017-04-24 22:23:14]

Ryan, please take a look. I have no clue what combinations would actually be
useful to test, so I threw something in as an example in order to test the
framework. 
Let me know if there is a more interesting field trial combination I should use

[Robert Ogden, 2017-04-24 22:23:31]

Description was changed from

==========
Add variations combinations testing framework
==========

to

==========
Add variations combinations testing framework

BUG=714827
==========

[RyanSturm, 2017-04-27 20:17:10]

Need to take another look, but I'd suggest that we land this as is (maybe change
current combinations), and then send out an email or doc to the team to gather
suggestions of what combinations to test.

[Robert Ogden, 2017-04-27 20:19:15]

On 2017/04/27 20:17:10, Ryan Sturm wrote:
> Need to take another look, but I'd suggest that we land this as is (maybe
change
> current combinations), and then send out an email or doc to the team to gather
> suggestions of what combinations to test.

That sgtm

[RyanSturm, 2017-04-27 21:46:39]

LGTM % comments, land and send out as email as previously discussed.

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
File tools/chrome_proxy/webdriver/variations_combinations.py (right):

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
tools/chrome_proxy/webdriver/variations_combinations.py:14: # EDIT BETWEEN THESE
LINES
Remove these comments.

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
tools/chrome_proxy/webdriver/variations_combinations.py:19: 'tests': [
Would it make sense to give a name to each test combination (dict instead of
array)? I'm torn if it adds any value.

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
tools/chrome_proxy/webdriver/variations_combinations.py:32:
"DataCompressionProxyLoFi.Enabled_Preview:effective_connection_type/4G"
These seem fine as a dummy example.

[Robert Ogden, 2017-04-28 15:48:47]

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
File tools/chrome_proxy/webdriver/variations_combinations.py (right):

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
tools/chrome_proxy/webdriver/variations_combinations.py:14: # EDIT BETWEEN THESE
LINES
On 2017/04/27 21:46:39, Ryan Sturm wrote:
> Remove these comments.

Done.

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
tools/chrome_proxy/webdriver/variations_combinations.py:19: 'tests': [
On 2017/04/27 21:46:39, Ryan Sturm wrote:
> Would it make sense to give a name to each test combination (dict instead of
> array)? I'm torn if it adds any value.

I think that's a great idea. It gives some documentation to the code. I'm not
sure how to plumb it into running the test, but I've added it anyways. I'll take
a look at it again once we've got some actual tests.

https://codereview.chromium.org/2843463004/diff/1/tools/chrome_proxy/webdrive...
tools/chrome_proxy/webdriver/variations_combinations.py:32:
"DataCompressionProxyLoFi.Enabled_Preview:effective_connection_type/4G"
On 2017/04/27 21:46:39, Ryan Sturm wrote:
> These seem fine as a dummy example.

Acknowledged.

[Robert Ogden, 2017-04-28 15:48:50]

The CQ bit was checked by robertogden@chromium.org

[Robert Ogden, 2017-04-28 15:48:51]

The patchset sent to the CQ was uploaded after l-g-t-m from
ryansturm@chromium.org
Link to the patchset: https://codereview.chromium.org/2843463004/#ps80001
(title: "fix merge issue 3")

[commit-bot: I haz the power, 2017-04-28 15:49:21]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-28 16:34:47]

CQ is committing da patch.

Bot data: {"patchset_id": 80001, "attempt_start_ts": 1493394530790450,
"parent_rev": "c79ffd81351cd9ade16c129ff4e7728e7275dfac", "commit_rev":
"fb645f073cb67786bb972c80edcde253f733b29c"}

[commit-bot: I haz the power, 2017-04-28 16:34:53]

Description was changed from

==========
Add variations combinations testing framework

BUG=714827
==========

to

==========
Add variations combinations testing framework

BUG=714827

Review-Url: https://codereview.chromium.org/2843463004
Cr-Commit-Position: refs/heads/master@{#468025}
Committed:
https://chromium.googlesource.com/chromium/src/+/fb645f073cb67786bb972c80edcd...
==========

[commit-bot: I haz the power, 2017-04-28 16:34:56]

Committed patchset #5 (id:80001) as
https://chromium.googlesource.com/chromium/src/+/fb645f073cb67786bb972c80edcd...