Create test expectations line automatically from failing test results

third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py

Index: third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py
diff --git a/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py b/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py
new file mode 100644
index 0000000000000000000000000000000000000000..d3588a33f0d6fd1401bae197d17b5034457182e7
--- /dev/null
+++ b/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py
@@ -0,0 +1,22 @@
+from webkitpy.w3c.update_w3c_test_expectations import *

[qyearsley, 2016/07/12 23:00:15]

Good start to the unit test :-) Remember to add that file header at the top.

[dcampb, 2016/07/15 00:24:13]

I know you were saying to refrain from using the * when importing, however my
script is not object oriented. Should I continue to use the star or just import
the functions. Or, somehow modify my script ? 

[qyearsley, 2016/07/15 16:42:47]

If you don't use * when importing, then your choices are:


1. Import the module itself, e.g. `from webkitpy.w3c import
update_w3c_test_expectations`

Then specific functions inside your class are referenced as
`update_w3c_test_expectations.update_platforms`.

If you feel that update_w3c_test_expectations is a bit too long, you can give it
an alias after importing, e.g. `from webkitpy.w3c import
update_w3c_test_expectations as update_module`.

Even though this is a little more verbose, it has the advantage that when
reading the tests, it's clear where all of the functions are defined.


2. If you're testing a relatively small number of names from a module, it might
be nicer to just import the names you're interested in, e.g. `from
webkitpy.w3c.update_w3c_test_expectations import create_line_list`.

+import unittest

[qyearsley, 2016/07/12 23:00:15]

Regarding the imports:

1. Wildcard imports are generally discouraged; we may also want to consider
importing the module with `from webkitpy.w3c import
update_w3c_test_expectations`.

2. Standard library imports (e.g. unittest) generally go first.

+
+
+class UpdateW3cTestExpectationsTest(unittest.TestCase):

[qyearsley, 2016/07/12 23:00:15]

W3c -> W3C

+
+    def test_merge_platforms(self):
+        example_one = {'foo/bar/man.html': {'one': {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}, 'two': {'expected': "FAIL", 'actual': 'PASS', 'bug': 'crbug.com/626703'}}}

[qyearsley, 2016/07/12 23:00:15]

These are much easier to read when formatted more like this:

example_one = {
    'foo/bar/man.html': {
        'one': {'expected': 'FAIL', 'actual': 'PASS', 'bug':
'crbug.com/626703'},
        'two': {'expected': "FAIL", 'actual': 'PASS', 'bug':
'crbug.com/626703'},
    }
}

+        example_two = {'foo/bar/man.html': {'one': {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}, 'two': {'expected': u'FAIL', 'actual': u'TIMEOUT', 'bug': 'crbug.com/626703'}, 'three': {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}}}
+        self.assertEqual(update_platforms(example_one), {'foo/bar/man.html':  {('two', 'one'): {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}}})
+        self.assertEqual(update_platforms(example_two), {'foo/bar/man.html':  {('three', 'one'): {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}, 'two': {'expected': u'FAIL', 'actual': u'TIMEOUT', 'bug': 'crbug.com/626703'}}})
+
+    def test__get_expectations(self):

[qyearsley, 2016/07/12 23:00:15]

test__ -> test_

[dcampb, 2016/07/15 00:24:13]

Should I be testing private functions? Or should I just test the function that
call them and mock their functionality?

[qyearsley, 2016/07/15 16:42:47]

You can test private functions :-)

If it's possible to test them by testing the function that calls them, then it's
better to do that, and you can skip testing the private functions directly.

When testing a private function named _foo, you don't need to name the test
method test__foo, you can just call it test_foo. In general,  test method names
only serves as a description of what's being tested, it doesn't necessarily need
to be of the "form test_<function>".

+        self.assertEqual(get_expectations({'expected': 'FAIL', 'actual': 'PASS'}), ['Pass'])
+        self.assertEqual(get_expectations({'expected': 'FAIL', 'actual': 'TIMEOUT'}), ['Timeout'])
+        self.assertEqual(get_expectations({'expected': 'TIMEOUT', 'actual': 'PASS'}), ['Pass', 'Timeout'])
+
+    def test_create_line_list(self):
+        example_one = {'foo/bar/man.html': {'one': {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}, 'two': {'expected': "FAIL", 'actual': 'PASS', 'bug': 'crbug.com/626703'}}}
+        example_two = {'foo/bar/man.html': {'one': {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}, 'two': {'expected': u'FAIL', 'actual': u'TIMEOUT', 'bug': 'crbug.com/626703'}, 'three': {'expected': u'FAIL', 'actual': u'PASS', 'bug': 'crbug.com/626703'}}}
+        self.assertEqual(create_line_list(example_one), ['crbug.com/626703 [ two ] foo/bar/man.html [ Pass ]', u'crbug.com/626703 [ one ] foo/bar/man.html [ Pass ]'])

[qyearsley, 2016/07/12 23:00:15]

Lines like this can also be made more readable by reformatting, e.g.

self.assertEqual(
    create_line_list(example_one),
    [
        'crbug.com/626703 [ two ] foo/bar/man.html [ Pass ]',
        'crbug.com/626703 [ one ] foo/bar/man.html [ Pass ]'
    ])

+        self.assertEqual(create_line_list(example_two), [u'crbug.com/626703 [ three ] foo/bar/man.html [ Pass ]', u'crbug.com/626703 [ two ] foo/bar/man.html [ Timeout ]', u'crbug.com/626703 [ one ] foo/bar/man.html [ Pass ]'])