DevTools: enable backwards-compatibility testing

DevTools: enable backwards-compatibility testing

This lays down the groundwork for doing backwards-compatibility testing
to ensure newer front-ends don't break backends supporting devtools protocol 1.2.

Because there will be diffs from the tip-of-tree baseline, I'm creating
a set of baselines at: 
devtools/tests/baseline/protocol-1.2/test-name...

Usage:
npm test -- --compat-protocol=1.2

BUG=682078
Review-Url: https://codereview.chromium.org/2648633002
Cr-Commit-Position: refs/heads/master@{#445538}
Committed: https://chromium.googlesource.com/chromium/src/+/a35b250bcc6b0afde5638c874d409b90e0f3d288

[chenwilliam, 2017-01-19 23:07:42]

Description was changed from

==========
DevTools: enable backwards-compatibility testing

This lays down the groundwork for doing backwards-compatibility testing
to ensure newer front-ends don't break backends supporting devtools protocol
1.2.

Because there will be diffs from the tip-of-tree baseline, I'm treating this
as a separate platform (e.g. linux-devtools-1.2) so it can have its own
platform-
specific expectations.

npm test -- --compat-protocol=1.2

BUG=682078
==========

to

==========
DevTools: enable backwards-compatibility testing

This lays down the groundwork for doing backwards-compatibility testing
to ensure newer front-ends don't break backends supporting devtools protocol
1.2.

Because there will be diffs from the tip-of-tree baseline, I'm treating this
as a separate platform (e.g. linux-devtools-1.2) so it can have its own
platform-
specific expectations.

Usage:
npm test -- --compat-protocol=1.2

BUG=682078
==========

[chenwilliam, 2017-01-19 23:08:00]

chenwilliam@chromium.org changed reviewers:
+ dgozman@chromium.org, dpranke@chromium.org, pfeldman@chromium.org

[chenwilliam, 2017-01-19 23:08:00]

@dpranke - I wanted to get your feedback on my usage of "platform-specific
expectations" for the layout test harness because it's a bit unconventional.

Please see the linked bug for background on what we're trying to achieve.

@pfeldman @dgozman - ptal. After this CL, I'll start going through the test
results and landing new baselines where needed and fixing timeouts.

https://codereview.chromium.org/2648633002/diff/1/third_party/WebKit/Source/d...
File third_party/WebKit/Source/devtools/front_end/protocol/InspectorBackend.js
(left):

https://codereview.chromium.org/2648633002/diff/1/third_party/WebKit/Source/d...
third_party/WebKit/Source/devtools/front_end/protocol/InspectorBackend.js:637:
if (messageObject.error && messageObject.error.code !==
Protocol.InspectorBackend._ConnectionClosedErrorCode &&
I removed this after discussing with @pfeldman because missing protocol methods
are being logged into the test results. Each caller that gracefully handles
missing protocol methods should have a customized log which will be recorded in
the test results.

Example:
error: Request Debugger.getPossibleBreakpoints failed.
{"code":-32601,"message":"'Debugger.getPossibleBreakpoints' wasn't found"}

[Dirk Pranke, 2017-01-20 02:14:40]

From the harness' point of view, this probably isn't the way I'd approach this
problem. 

Ideally you'd have a separate test step that just ran the set of tests, pointed
at your old binary, and pointed at a different set of expectation files. You
should be able to all of those things w/ just different command line flags and
not need to change any code; look into the --build-directory, --target,
--additional-expectations, and --additional-platform-directory flags.

[dgozman, 2017-01-20 17:52:09]

I don't have an opinion on infrastructure here.

https://codereview.chromium.org/2648633002/diff/1/third_party/WebKit/Source/d...
File third_party/WebKit/Source/devtools/front_end/protocol/InspectorBackend.js
(left):

https://codereview.chromium.org/2648633002/diff/1/third_party/WebKit/Source/d...
third_party/WebKit/Source/devtools/front_end/protocol/InspectorBackend.js:637:
if (messageObject.error && messageObject.error.code !==
Protocol.InspectorBackend._ConnectionClosedErrorCode &&
On 2017/01/19 23:08:00, chenwilliam wrote:
> I removed this after discussing with @pfeldman because missing protocol
methods
> are being logged into the test results. Each caller that gracefully handles
> missing protocol methods should have a customized log which will be recorded
in
> the test results.
> 
> Example:
> error: Request Debugger.getPossibleBreakpoints failed.
> {"code":-32601,"message":"'Debugger.getPossibleBreakpoints' wasn't found"}

Let's land this separately:
- I'm worried we will miss some new issues due to no logging;
- removing this allows more cleanup on error codes.

[chenwilliam, 2017-01-20 20:16:39]

Description was changed from

==========
DevTools: enable backwards-compatibility testing

This lays down the groundwork for doing backwards-compatibility testing
to ensure newer front-ends don't break backends supporting devtools protocol
1.2.

Because there will be diffs from the tip-of-tree baseline, I'm treating this
as a separate platform (e.g. linux-devtools-1.2) so it can have its own
platform-
specific expectations.

Usage:
npm test -- --compat-protocol=1.2

BUG=682078
==========

to

==========
DevTools: enable backwards-compatibility testing

This lays down the groundwork for doing backwards-compatibility testing
to ensure newer front-ends don't break backends supporting devtools protocol
1.2.

Because there will be diffs from the tip-of-tree baseline, I'm creating
a set of baselines at: 
devtools/tests/baseline/protocol-1.2/test-name...

Usage:
npm test -- --compat-protocol=1.2

BUG=682078
==========

[chenwilliam, 2017-01-20 20:19:22]

Thanks for the pointer Dirk - I've removed the changes to the test harness.

@dgozman - I've undone the removal of error logging - PTAL.

https://codereview.chromium.org/2648633002/diff/1/third_party/WebKit/Source/d...
File third_party/WebKit/Source/devtools/front_end/protocol/InspectorBackend.js
(left):

https://codereview.chromium.org/2648633002/diff/1/third_party/WebKit/Source/d...
third_party/WebKit/Source/devtools/front_end/protocol/InspectorBackend.js:637:
if (messageObject.error && messageObject.error.code !==
Protocol.InspectorBackend._ConnectionClosedErrorCode &&
On 2017/01/20 at 17:52:09, dgozman wrote:
> On 2017/01/19 23:08:00, chenwilliam wrote:
> > I removed this after discussing with @pfeldman because missing protocol
methods
> > are being logged into the test results. Each caller that gracefully handles
> > missing protocol methods should have a customized log which will be recorded
in
> > the test results.
> > 
> > Example:
> > error: Request Debugger.getPossibleBreakpoints failed.
> > {"code":-32601,"message":"'Debugger.getPossibleBreakpoints' wasn't found"}
> 
> Let's land this separately:
> - I'm worried we will miss some new issues due to no logging;
> - removing this allows more cleanup on error codes.

OK, removing from CL. What kind of cleanup on error codes do you want? (I can do
it?)

[dgozman, 2017-01-23 20:03:09]

lgtm

[chenwilliam, 2017-01-23 20:28:03]

The CQ bit was checked by chenwilliam@chromium.org

[commit-bot: I haz the power, 2017-01-23 20:28:43]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-23 20:30:53]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-23 20:30:55]

Try jobs failed on following builders:
  ios-device on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device/builds...)
  ios-device-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device-xcode-...)
  ios-simulator on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...)
  ios-simulator-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator-xco...)

[chenwilliam, 2017-01-23 21:51:34]

The CQ bit was checked by chenwilliam@chromium.org

[chenwilliam, 2017-01-23 21:51:34]

The patchset sent to the CQ was uploaded after l-g-t-m from dgozman@chromium.org
Link to the patchset: https://codereview.chromium.org/2648633002/#ps60001
(title: "nits")

[commit-bot: I haz the power, 2017-01-23 21:52:18]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-23 23:16:44]

CQ is committing da patch.

Bot data: {"patchset_id": 60001, "attempt_start_ts": 1485208294172940,
"parent_rev": "063604c41cf636c5be9575f501e5fa5e6e539a33", "commit_rev":
"a35b250bcc6b0afde5638c874d409b90e0f3d288"}

[commit-bot: I haz the power, 2017-01-23 23:17:14]

Description was changed from

==========
DevTools: enable backwards-compatibility testing

This lays down the groundwork for doing backwards-compatibility testing
to ensure newer front-ends don't break backends supporting devtools protocol
1.2.

Because there will be diffs from the tip-of-tree baseline, I'm creating
a set of baselines at: 
devtools/tests/baseline/protocol-1.2/test-name...

Usage:
npm test -- --compat-protocol=1.2

BUG=682078
==========

to

==========
DevTools: enable backwards-compatibility testing

This lays down the groundwork for doing backwards-compatibility testing
to ensure newer front-ends don't break backends supporting devtools protocol
1.2.

Because there will be diffs from the tip-of-tree baseline, I'm creating
a set of baselines at: 
devtools/tests/baseline/protocol-1.2/test-name...

Usage:
npm test -- --compat-protocol=1.2

BUG=682078

Review-Url: https://codereview.chromium.org/2648633002
Cr-Commit-Position: refs/heads/master@{#445538}
Committed:
https://chromium.googlesource.com/chromium/src/+/a35b250bcc6b0afde5638c874d40...
==========

[commit-bot: I haz the power, 2017-01-23 23:17:17]

Committed patchset #4 (id:60001) as
https://chromium.googlesource.com/chromium/src/+/a35b250bcc6b0afde5638c874d40...