Make it possible to enable/disable specific ppapi tests. Migrate PostMessage tests.

Make it possible to enable/disable specific ppapi tests. Migrate PostMessage tests.

Most of these files were changed by a sed script, so it's not as bad as it looks.

The testcase attribute now can include a 'filter'. If it's omitted, everything works the same as before. This way we can migrate tests over bit-by-bit if we want to. We can also still run the tests manually the same way as before.

This only runs PostMessage testss the new way, and re-enables all oop PostMessage tests that pass on Windows. I can do the other tests in this CL if desired, but it might be easier to land in a few pieces.

BUG=102885, 95557
TEST=N/A

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=109114

[darin (slow to review), 2011-11-07 23:33:11]

when checking the filter against a test name, maybe we should just be very
specific about testing that the test name ends with "_" + filter?  that way we
avoid false hits.

http://codereview.chromium.org/8477015/diff/1/ppapi/tests/testing_instance.cc
File ppapi/tests/testing_instance.cc (right):

http://codereview.chromium.org/8477015/diff/1/ppapi/tests/testing_instance.cc...
ppapi/tests/testing_instance.cc:171: std::string
TestingInstance::FilterForTestName(const std::string& name) {
hmm... maybe calling this a filter is a bit too generic, when in reality it is
just the name of the test within the test group.  maybe it would help to just
have a good name for these two terms.  maybe look at what gtest calls them
(group == gtest file)?

[dmichael (off chromium), 2011-11-08 16:29:34]

On 2011/11/07 23:33:11, darin wrote:
> when checking the filter against a test name, maybe we should just be very
> specific about testing that the test name ends with "_" + filter?  that way we
> avoid false hits.
I wanted to make it possible to still run the tests manually in a group as we
have in the past. Otherwise, I have to do some more work to enumerate/list all
the tests for each test case. With the patch as-is, the same old URLs work, so
if you navigate to:
http://testserver/ppapi/tests/test_case.html
All the test cases are listed. When you click on one and go to, e.g.:
http://testserver/ppapi/tests/test_case.html?testcase=PostMessage
It still runs & displays all tests for that test case.

This also gives us a nice way to transition the tests piece by piece.
> 
> http://codereview.chromium.org/8477015/diff/1/ppapi/tests/testing_instance.cc
> File ppapi/tests/testing_instance.cc (right):
> 
>
http://codereview.chromium.org/8477015/diff/1/ppapi/tests/testing_instance.cc...
> ppapi/tests/testing_instance.cc:171: std::string
> TestingInstance::FilterForTestName(const std::string& name) {
> hmm... maybe calling this a filter is a bit too generic, when in reality it is
> just the name of the test within the test group.  maybe it would help to just
> have a good name for these two terms.  maybe look at what gtest calls them
> (group == gtest file)?
As written, it really is a filter (albeit a dumb one); each test case will run
all tests whose name contains the filter as a substring. If you provide no '_',
the empty string means 'run all tests'. This does give us a weird in-between
capability we don't really need. E.g.:
http://testserver/ppapi/tests/test_case.html?testcase=PostMessage_Send
Will run these two tests:
PostMessage_SendInInit
PostMessage_SendingData
This could be problematic if we have test names which are substrings of other
test names.

Would you rather I change up the approach a bit? I like having the capability to
run all tests in a test case, but I could easily make it just do 'all tests' or
'1 test' for each test case.

[darin (slow to review), 2011-11-08 16:35:03]

On Tue, Nov 8, 2011 at 8:29 AM, <dmichael@chromium.org> wrote:

> On 2011/11/07 23:33:11, darin wrote:
>
>> when checking the filter against a test name, maybe we should just be very
>> specific about testing that the test name ends with "_" + filter?  that
>> way we
>> avoid false hits.
>>
> I wanted to make it possible to still run the tests manually in a group as
> we
> have in the past. Otherwise, I have to do some more work to enumerate/list
> all
> the tests for each test case. With the patch as-is, the same old URLs
> work, so
> if you navigate to:
>
http://testserver/ppapi/tests/**test_case.html<http://testserver/ppapi/tests/...
> All the test cases are listed. When you click on one and go to, e.g.:
>
http://testserver/ppapi/tests/**test_case.html?testcase=**PostMessage<http://...
> It still runs & displays all tests for that test case.
>
> This also gives us a nice way to transition the tests piece by piece.
>
>
>  http://codereview.chromium.**org/8477015/diff/1/ppapi/**
>>
tests/testing_instance.cc<http://codereview.chromium.org/8477015/diff/1/ppapi/tests/testing_instance.cc>
>> File ppapi/tests/testing_instance.**cc (right):
>>
>
>
> http://codereview.chromium.**org/8477015/diff/1/ppapi/**
>
tests/testing_instance.cc#**newcode171<http://codereview.chromium.org/8477015/diff/1/ppapi/tests/testing_instance.cc#newcode171>
>
>> ppapi/tests/testing_instance.**cc:171: std::string
>> TestingInstance::**FilterForTestName(const std::string& name) {
>> hmm... maybe calling this a filter is a bit too generic, when in reality
>> it is
>> just the name of the test within the test group.  maybe it would help to
>> just
>> have a good name for these two terms.  maybe look at what gtest calls them
>> (group == gtest file)?
>>
> As written, it really is a filter (albeit a dumb one); each test case will
> run
> all tests whose name contains the filter as a substring. If you provide no
> '_',
> the empty string means 'run all tests'. This does give us a weird
> in-between
> capability we don't really need. E.g.:
>
http://testserver/ppapi/tests/**test_case.html?testcase=**PostMessage_Send<ht...
> Will run these two tests:
> PostMessage_SendInInit
> PostMessage_SendingData
> This could be problematic if we have test names which are substrings of
> other
> test names.
>
> Would you rather I change up the approach a bit? I like having the
> capability to
> run all tests in a test case, but I could easily make it just do 'all
> tests' or
> '1 test' for each test case.
>
>
Substring matching is fine.  It might make things more predictable if we
used a "startswith" or "endswith" filter though.
-Darin



>
http://codereview.chromium.**org/8477015/<http://codereview.chromium.org/8477...
>

[dmichael (off chromium), 2011-11-08 17:39:54]

I realized we have a pattern where we'll run tests with and without 'force
async' CompletionCallbacks:
TestCase_TestName
TestCase_TestNameForceAsync

So the filtering won't work for those. I went ahead and made it so that RUN_TEST
will only run if (a) the filter is empty or (b) the filter is an exact match.

[darin (slow to review), 2011-11-08 21:02:26]

LGTM