Re-enable the DB perf tests and lower the number of iterations per...

Re-enable the DB perf tests and lower the number of iterations per
test from 10 to 5, to make sure all tests finish in less than 10
minutes (seems to be a problem only on the Vista perf bot).

TEST=none
BUG=none


Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=42284

[darin (slow to review), 2010-03-22 06:56:49]

OK, but...

http://codereview.chromium.org/1089006/diff/1/2
File chrome/test/page_cycler/page_cycler_test.cc (right):

http://codereview.chromium.org/1089006/diff/1/2#newcode39
chrome/test/page_cycler/page_cycler_test.cc:39: #define DATABASE_TEST_ITERATIONS
5
won't this increase variance?  or is the variance level acceptable?  or maybe we
won't know until the tests have been running for a while?  have you considered
getting the buildbot changed to allow the test to run longer?

[Paweł Hajdan Jr., 2010-03-22 07:24:19]

Drive-by.

You can also try printing some output after each iteration (and flush stdout to
be sure). I think that should prevent the buildbot from killing the test process
due to lack of output. You can ask Nicolas Sylvain to be sure.

[dumi, 2010-03-22 20:50:23]

On 2010/03/22 06:56:49, darin wrote:
> OK, but...
> 
> http://codereview.chromium.org/1089006/diff/1/2
> File chrome/test/page_cycler/page_cycler_test.cc (right):
> 
> http://codereview.chromium.org/1089006/diff/1/2#newcode39
> chrome/test/page_cycler/page_cycler_test.cc:39: #define
DATABASE_TEST_ITERATIONS
> 5
> won't this increase variance?  or is the variance level acceptable?  or maybe
we
> won't know until the tests have been running for a while?  have you considered
> getting the buildbot changed to allow the test to run longer?

i ran these tests many times on my local machine, and they ran a few times on
various perf bots too, and looking at the data we got, i believe that running
these tests 5 times would yield pretty much the same results as running them 10
times.

pawel: i could try to print partial results. however, in that case, the DB tests
would take over an hour to run on the vista perf bot. we probably don't want
that, and since we can reduce the number of iterations without skewing the data,
i think that might be a better solution here.

[darin (slow to review), 2010-03-22 20:53:01]

On Mon, Mar 22, 2010 at 1:50 PM, <dumi@chromium.org> wrote:

> On 2010/03/22 06:56:49, darin wrote:
>
>> OK, but...
>>
>
>  http://codereview.chromium.org/1089006/diff/1/2
>> File chrome/test/page_cycler/page_cycler_test.cc (right):
>>
>
>  http://codereview.chromium.org/1089006/diff/1/2#newcode39
>> chrome/test/page_cycler/page_cycler_test.cc:39: #define
>>
> DATABASE_TEST_ITERATIONS
>
>> 5
>> won't this increase variance?  or is the variance level acceptable?  or
>> maybe
>>
> we
>
>> won't know until the tests have been running for a while?  have you
>> considered
>> getting the buildbot changed to allow the test to run longer?
>>
>
> i ran these tests many times on my local machine, and they ran a few times
> on
> various perf bots too, and looking at the data we got, i believe that
> running
> these tests 5 times would yield pretty much the same results as running
> them 10
> times.
>
> pawel: i could try to print partial results. however, in that case, the DB
> tests
> would take over an hour to run on the vista perf bot.


why is that the case?  it doesn't seem obvious to me.

-darin




> we probably don't want
> that, and since we can reduce the number of iterations without skewing the
> data,
> i think that might be a better solution here.
>
>
> http://codereview.chromium.org/1089006
>

[darin (slow to review), 2010-03-22 22:21:01]

http://codereview.chromium.org/1089006/diff/1/2
File chrome/test/page_cycler/page_cycler_test.cc (right):

http://codereview.chromium.org/1089006/diff/1/2#newcode39
chrome/test/page_cycler/page_cycler_test.cc:39: #define DATABASE_TEST_ITERATIONS
5
On 2010/03/22 06:56:49, darin wrote:
> won't this increase variance?  or is the variance level acceptable?  or maybe
we
> won't know until the tests have been running for a while?  have you considered
> getting the buildbot changed to allow the test to run longer?

It would be good to at least file a bug about the vista perf bot and reference
that bug here.  It seems like we should try to understand the root cause of the
vista perf bot since the same issue could impact end users.

[dumi, 2010-03-22 22:40:14]

On 2010/03/22 22:21:01, darin wrote:
> http://codereview.chromium.org/1089006/diff/1/2
> File chrome/test/page_cycler/page_cycler_test.cc (right):
> 
> http://codereview.chromium.org/1089006/diff/1/2#newcode39
> chrome/test/page_cycler/page_cycler_test.cc:39: #define
DATABASE_TEST_ITERATIONS
> 5
> On 2010/03/22 06:56:49, darin wrote:
> > won't this increase variance?  or is the variance level acceptable?  or
maybe
> we
> > won't know until the tests have been running for a while?  have you
considered
> > getting the buildbot changed to allow the test to run longer?
> 
> It would be good to at least file a bug about the vista perf bot and reference
> that bug here.  It seems like we should try to understand the root cause of
the
> vista perf bot since the same issue could impact end users.

if we used 10 iterations, the tests would take more than an hour because 1
iteration takes about 2 minutes, so 1 test would take about 20 minutes, and we
have 4-5 tests that could take this much time to run.

opened http://code.google.com/p/chromium/issues/detail?id=38962 to keep track of
this problem.

[chase, 2010-03-22 22:51:23]

thanks for filing 38962.  do you see this as a stopgap until the vista perf
issue is sorted out?  that is, after that system is fixed, will you revert this
change (or at least return the DB test to 10 iterations)?

[chase, 2010-03-22 22:52:13]

lgtm, btw

[dumi, 2010-03-22 23:04:28]

On 2010/03/22 22:51:23, chase wrote:
> thanks for filing 38962.  do you see this as a stopgap until the vista perf
> issue is sorted out?  that is, after that system is fixed, will you revert
this
> change (or at least return the DB test to 10 iterations)?

i'd probably revert this patch, to keep things consistent across all page_cycler
tests. but we can decide what we want to do when the issue on the vista perf bot
is resolved.