NOT FOR LANDING: How do we add unit tests for bindings?

NOT FOR LANDING: How do we add unit tests for bindings?

BUG=

[dglazkov, 2016-12-05 05:32:44]

dglazkov@chromium.org changed reviewers:
+ dpranke@chromium.org

[dglazkov, 2016-12-05 05:38:08]

Dirk, would love your thoughts on this. My intent is to add unit tests to the
bindings code, which currently lives in Source/bindings/scripts.

There are few of ways of doing this:

1) Glom another invocation of typ onto Tools/Scripts/run-bindings-tests. Pros:
easy. Cons: it's not exactly the right thing, since I am mixing end-to-end tests
with the unit tests.

2) Tack that invocation at the end of Tools/Scripts/test-webkitpy. Pros: easy.
Cons: it's kind of ugly, because now there are two distinct runs in that file,
and one of them is not technically over webkitpy.

3) Migrate IDL compiler to webkitpy. Pros: unit tests start working
automatically. Cons: not sure if that's the right thing, and slightly harder to
do.

4) Create another test runner, like Tools/Scripts/test-bindings, then add it to
run alongside test-webkitpy. Pros: seems logical. Cons: have to mess with
recipes, which seems hard.

I provided an illustration of how the #2 option would look like. I would
appreciate your thoughts :)

[dglazkov, 2016-12-05 05:48:30]

+bashi, yukishiino, haraken because they probably have opinions, too.

[Yuki, 2016-12-05 06:47:22]

On 2016/12/05 05:48:30, dglazkov wrote:
> +bashi, yukishiino, haraken because they probably have opinions, too.

I think this is a problem how to invoke the bindings unittests, which you're
going to create newly.  Then, as long as it's not a hard work to change or
revert later, I think we can start with an easy way at first.  After we get
plenty of unittests worth running elegantly, we may want to invoke the unittests
elegantly.

So, my opinion is that I'd recommend to start with an easy way rather than a
correct/smart/elegant way, as long as it's not hard to change later.  I will
accept a bit ugly hack as the first step (TODO comments are welcome).

[bashi, 2016-12-05 07:21:14]

On 2016/12/05 05:38:08, dglazkov wrote:
> Dirk, would love your thoughts on this. My intent is to add unit tests to the
> bindings code, which currently lives in Source/bindings/scripts.
> 
> There are few of ways of doing this:
> 
> 1) Glom another invocation of typ onto Tools/Scripts/run-bindings-tests. Pros:
> easy. Cons: it's not exactly the right thing, since I am mixing end-to-end
tests
> with the unit tests.
> 
> 2) Tack that invocation at the end of Tools/Scripts/test-webkitpy. Pros: easy.
> Cons: it's kind of ugly, because now there are two distinct runs in that file,
> and one of them is not technically over webkitpy.
> 
> 3) Migrate IDL compiler to webkitpy. Pros: unit tests start working
> automatically. Cons: not sure if that's the right thing, and slightly harder
to
> do.
> 
> 4) Create another test runner, like Tools/Scripts/test-bindings, then add it
to
> run alongside test-webkitpy. Pros: seems logical. Cons: have to mess with
> recipes, which seems hard.
> 
> I provided an illustration of how the #2 option would look like. I would
> appreciate your thoughts :)

Thank you for raising this issue. It's definitely great to have unit tests for
bindings code generator. I slightly prefer migrating some portions of
bindings/scripts to other places (maybe webkitpy or tools/idl_parser) not only
for unit tests but also reusing IDL parser. When I wrote a simple web app that
shows diffs of IDLs since we forked from WebKit[1], I used IDL parser with very
hacky workaround. If we can use IDL parser outside Blink, it will help us create
other tools like "show IDL diffs between versions".

Just my 2 cents.
[1] https://github.com/bashi/blink-misc/tree/master/diff

[Dirk Pranke, 2016-12-06 02:14:38]

Whatever you are running should be based on top of typ, since that provides all
sorts of common logic that we use to run things on the builders.

I suppose that means that you should use approach (1) -- or at least that might
be the easiest thing to do -- but (3) would also work.

I wouldn't worry too much about intermixing "unit" and "end to end" tests;
test-webkitpy does that already as well.

[dglazkov, 2016-12-06 05:01:28]

Follow-up here: https://codereview.chromium.org/2554503004