Additional unit test for audit.js: failure cases and logging examples.

third_party/WebKit/LayoutTests/webaudio/unit-tests/audit-failures-expected.txt

Index: third_party/WebKit/LayoutTests/webaudio/unit-tests/audit-failures-expected.txt
diff --git a/third_party/WebKit/LayoutTests/webaudio/unit-tests/audit-failures-expected.txt b/third_party/WebKit/LayoutTests/webaudio/unit-tests/audit-failures-expected.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c711366902dd21ff6f87a3e0be7d441f5b78cf26
--- /dev/null
+++ b/third_party/WebKit/LayoutTests/webaudio/unit-tests/audit-failures-expected.txt
@@ -0,0 +1,46 @@
+This is a testharness.js-based test.
+PASS # AUDIT TASK RUNNER STARTED. 
+PASS > [basic-failure] Testing basic assertion failures. 
+FAIL X Setting foo to bar *INCORRECTLY* threw ReferenceError: "bar is not defined". assert_true: expected true got false
+FAIL X function () { var foo = 0; } did *NOT* throw an exception. assert_true: expected true got false
+FAIL X 3 < 5 is *NOT* true. assert_true: expected true got false
+FAIL X true is *NOT* false. assert_true: expected true got false
+FAIL X 1 is *NOT* equal to 2. assert_true: expected true got false
+FAIL X 1 should *NOT* be equal to 1. assert_true: expected true got false
+FAIL X number is *NOT* equal to string. assert_true: expected true got false
+FAIL X 1 is *NOT* greater than 2. assert_true: expected true got false
+FAIL X 1 is *NOT* greater than or equal to 2. assert_true: expected true got false
+FAIL X 2 is *NOT* less than 1. assert_true: expected true got false
+FAIL X 2 is *NOT* less than or equal to 1. assert_true: expected true got false
+FAIL X Start OAC rendering rejected *INCORRECTLY* with TypeError: Failed to execute 'decodeAudioData' on 'BaseAudioContext': 1 argument required, but only 0 present.. assert_true: expected true got false
+FAIL X undefined resolved *INCORRECTLY*. assert_true: expected true got false

[Raymond Toy, 2016/12/09 22:10:26]

This test needs a better description because "undefined" has no meaning in the
context of this message.  How does "undefined" resolve?

[hongchan, 2016/12/19 22:14:06]

In this case, the author needs to provide the description. I missed it. We
decided not to generate the description automatically so it simply prints the
evaluated value here.

+FAIL < [basic-failure] 13 out of 13 assertions were failed. assert_true: expected true got false
+PASS > [numerical-failures] Testing numerical assertion failures. 
+FAIL X 0 is *NOT* 0.1 within an error of 0: 0 with error of 0. assert_true: expected true got false

[Raymond Toy, 2016/12/09 22:10:26]

This message looks wrong: "0 with error of 0".  It ought to be rephrased in some
way.

[hongchan, 2016/12/19 22:14:05]

Can you suggest something? This error message was written by you a while ago.

+FAIL X [1,8,11,18,6,5,5,5,123,49] contains 7 values that are *NOT* equal to 5: 

[Raymond Toy, 2016/12/09 22:10:26]

This error message isn't the best because it's hard to tell that you're really
expecting 5 for all values.  Can we rephrase that?

[hongchan, 2016/12/19 22:14:06]

Done.

+	Index	Actual
+	[0]	1
+	[1]	8
+	[2]	11
+	[3]	18
+	...and 3 more errors. assert_true: expected true got false
+FAIL X [1,8,11,18,6,5,5,5,123,49] contains 3 values that are *NOT* equal to [1,8,10,18,6,5,6,5,123,48]: 

[Raymond Toy, 2016/12/09 22:10:26]

This is badly phrased as well.  Perhaps it's better to say:

[...] expected to be equal to [...] but differs in 3 places:

[hongchan, 2016/12/19 22:14:06]

Done.

+	Index	Actual			Expected
+	[2]	1.1000000000000000e+1	1.0000000000000000e+1
+	[6]	5.0000000000000000e+0	6.0000000000000000e+0
+	[9]	4.9000000000000000e+1	4.8000000000000000e+1 assert_true: expected true got false
+FAIL X [1,1,1,1,2,2,3,3,3] contains an *UNEXPECTED* value of 3 at index 6. assert_true: expected true got false

[Raymond Toy, 2016/12/09 22:10:26]

This could be better phrased too because you can't tell what the actual output
was supposed to be. Maybe:

[...] expected to be [1..., 3..., 2...] but got 3 at index 6.

And shouldn't the error message say that 2 was unexpected at index 4?

[hongchan, 2016/12/19 22:14:05]

So we want the first instance of error. Right? I need to think about the
algorithm for that.

+FAIL X [0.5,0.5,0.55,0.5,0.45,0.5] has a *GLITCH* at index 5 of size 0.04999999999999999. assert_true: expected true got false
+FAIL X [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.9] does *NOT* equal [0.7,0.6,0.5,0.4,0.3,0.2,0.1] with an element-wise tolerance of {"absoluteThreshold":0.02,"relativeThreshold":0.1}.
+	Index	Actual			Expected		AbsError		RelError		Test threshold
+	[0]	1.0000000000000001e-1	6.9999999999999996e-1	5.9999999999999998e-1	8.5714285714285721e-1	6.9999999999999993e-2
+	[1]	2.0000000000000001e-1	5.9999999999999998e-1	3.9999999999999997e-1	6.6666666666666663e-1	5.9999999999999998e-2
+	[2]	2.9999999999999999e-1	5.0000000000000000e-1	2.0000000000000001e-1	4.0000000000000002e-1	5.0000000000000003e-2
+	[4]	5.0000000000000000e-1	2.9999999999999999e-1	2.0000000000000001e-1	6.6666666666666674e-1	2.9999999999999999e-2
+	[5]	5.9999999999999998e-1	2.0000000000000001e-1	3.9999999999999997e-1	1.9999999999999998e+0	2.0000000000000004e-2
+	Max AbsError of 5.9999999999999998e-1 at index of 0.
+	Max RelError of 5.9999999999999991e+0 at index of 6. assert_true: expected true got false
+FAIL < [numerical-failures] 6 out of 6 assertions were failed. assert_true: expected true got false
+FAIL # AUDIT TASK RUNNER FINISHED: 2 out of 2 tasks were failed. assert_true: expected true got false
+Harness: the test ran to completion.
+