Make test262 test runner check for which exception is thrown

Make test262 test runner check for which exception is thrown

test262 "negative" test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.

Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class.
It invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.

Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.

BUG=v8:4803
R=machenbach
LOG=Y

Committed: https://crrev.com/f3568ca4005fb81431f74175715f34a460bba765
Cr-Commit-Position: refs/heads/master@{#34763}

[Dan Ehrenberg, 2016-03-03 23:55:21]

The CQ bit was checked by littledan@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-03-03 23:55:30]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1766503002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1766503002/1

[Dan Ehrenberg, 2016-03-03 23:58:51]

Description was changed from

==========
Make test262 test runner check for which exception is thrown

test262 negative test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.
Previously, the test262 test runner ...
==========

to

==========
Make test262 test runner check for which exception is thrown

test262 negative test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.
Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class, and
it invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.
Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.

BUG=v8:4803
R=machenbach
LOG=Y
==========

[Dan Ehrenberg, 2016-03-03 23:58:51]

littledan@chromium.org changed reviewers:
+ machenbach@chromium.org

[commit-bot: I haz the power, 2016-03-04 00:07:34]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-03-04 00:07:35]

Dry run: Try jobs failed on following builders:
  v8_linux64_avx2_rel on tryserver.v8 (JOB_FAILED,
http://build.chromium.org/p/tryserver.v8/builders/v8_linux64_avx2_rel/builds/...)

[Michael Achenbach, 2016-03-10 12:16:31]

https://codereview.chromium.org/1766503002/diff/1/test/message/testcfg.py
File test/message/testcfg.py (right):

https://codereview.chromium.org/1766503002/diff/1/test/message/testcfg.py#new...
test/message/testcfg.py:99: testpath = testcase.testpath
testcase.path

https://codereview.chromium.org/1766503002/diff/1/test/webkit/testcfg.py
File test/webkit/testcfg.py (right):

https://codereview.chromium.org/1766503002/diff/1/test/webkit/testcfg.py#newc...
test/webkit/testcfg.py:123: file_name = os.path.join(self.root,
testcase.testpath) + "-expected.txt"
testcase.path

[Dan Ehrenberg, 2016-03-10 19:55:40]

The CQ bit was checked by littledan@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-03-10 19:55:47]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1766503002/20001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1766503002/20001

[Dan Ehrenberg, 2016-03-10 20:08:19]

PTAL

https://codereview.chromium.org/1766503002/diff/1/test/message/testcfg.py
File test/message/testcfg.py (right):

https://codereview.chromium.org/1766503002/diff/1/test/message/testcfg.py#new...
test/message/testcfg.py:99: testpath = testcase.testpath
On 2016/03/10 at 12:16:31, Michael Achenbach wrote:
> testcase.path

Thanks, fixed

https://codereview.chromium.org/1766503002/diff/1/test/webkit/testcfg.py
File test/webkit/testcfg.py (right):

https://codereview.chromium.org/1766503002/diff/1/test/webkit/testcfg.py#newc...
test/webkit/testcfg.py:123: file_name = os.path.join(self.root,
testcase.testpath) + "-expected.txt"
On 2016/03/10 at 12:16:31, Michael Achenbach wrote:
> testcase.path

Fixed

[commit-bot: I haz the power, 2016-03-10 20:09:44]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-03-10 20:09:45]

Dry run: Try jobs failed on following builders:
  v8_linux_arm_rel on tryserver.v8 (JOB_FAILED,
http://build.chromium.org/p/tryserver.v8/builders/v8_linux_arm_rel/builds/15154)

[Dan Ehrenberg, 2016-03-10 20:19:00]

The CQ bit was checked by littledan@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-03-10 20:19:11]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1766503002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1766503002/40001

[commit-bot: I haz the power, 2016-03-10 23:19:41]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-03-10 23:19:42]

Dry run: This issue passed the CQ dry run.

[Michael Achenbach, 2016-03-11 07:57:31]

lgtm - maybe split CL desc into 2-3 paragraphs for readability.

https://codereview.chromium.org/1766503002/diff/40001/test/test262/testcfg.py
File test/test262/testcfg.py (right):

https://codereview.chromium.org/1766503002/diff/40001/test/test262/testcfg.py...
test/test262/testcfg.py:182: for line in output.stdout.split("\n")[::-1]:
nit - readability: Maybe extract these four lines into their own method:
def _ExceptionFromOutput(output)
 for line in output.stdout.split("\n")[::-1]:
   if line and not line[0].isspace() and ":" in line:
     return line.split(":")[0]

Then inline the method call.

[Dan Ehrenberg, 2016-03-14 20:53:53]

Description was changed from

==========
Make test262 test runner check for which exception is thrown

test262 negative test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.
Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class, and
it invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.
Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.

BUG=v8:4803
R=machenbach
LOG=Y
==========

to

==========
Make test262 test runner check for which exception is thrown

test262 "negative" test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.

Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class.
It invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.

Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.

BUG=v8:4803
R=machenbach
LOG=Y
==========

[Dan Ehrenberg, 2016-03-14 21:00:05]

https://codereview.chromium.org/1766503002/diff/40001/test/test262/testcfg.py
File test/test262/testcfg.py (right):

https://codereview.chromium.org/1766503002/diff/40001/test/test262/testcfg.py...
test/test262/testcfg.py:182: for line in output.stdout.split("\n")[::-1]:
On 2016/03/11 at 07:57:31, Michael Achenbach wrote:
> nit - readability: Maybe extract these four lines into their own method:
> def _ExceptionFromOutput(output)
>  for line in output.stdout.split("\n")[::-1]:
>    if line and not line[0].isspace() and ":" in line:
>      return line.split(":")[0]
> 
> Then inline the method call.

Done

[Dan Ehrenberg, 2016-03-14 21:00:12]

The CQ bit was checked by littledan@chromium.org

[Dan Ehrenberg, 2016-03-14 21:00:12]

The patchset sent to the CQ was uploaded after l-g-t-m from
machenbach@chromium.org
Link to the patchset: https://codereview.chromium.org/1766503002/#ps60001
(title: "Better factoring")

[commit-bot: I haz the power, 2016-03-14 21:00:16]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1766503002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1766503002/60001

[commit-bot: I haz the power, 2016-03-14 21:19:17]

Committed patchset #4 (id:60001)

[commit-bot: I haz the power, 2016-03-14 21:20:45]

Description was changed from

==========
Make test262 test runner check for which exception is thrown

test262 "negative" test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.

Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class.
It invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.

Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.

BUG=v8:4803
R=machenbach
LOG=Y
==========

to

==========
Make test262 test runner check for which exception is thrown

test262 "negative" test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.

Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class.
It invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.

Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.

BUG=v8:4803
R=machenbach
LOG=Y

Committed: https://crrev.com/f3568ca4005fb81431f74175715f34a460bba765
Cr-Commit-Position: refs/heads/master@{#34763}
==========

[commit-bot: I haz the power, 2016-03-14 21:20:46]

Patchset 4 (id:??) landed as
https://crrev.com/f3568ca4005fb81431f74175715f34a460bba765
Cr-Commit-Position: refs/heads/master@{#34763}