I've addressed all the changes. My main remaining problem is how to deal with glut. ...
9 years, 9 months ago
(2011-03-30 05:26:49 UTC)
#3
I've addressed all the changes. My main remaining problem is how to deal with
glut. I really want to use the same version of glut as ebuild. This means for
now I can't get rid of SYSROOT?!
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
File client/site_tests/graphics_Piglit/control (right):
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
client/site_tests/graphics_Piglit/control:15: TEST_CLASS = "graphics"
On 2011/03/26 00:58:57, dalec wrote:
> Mixed quotes.
Done.
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
File client/site_tests/graphics_Piglit/graphics_Piglit.py (right):
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
client/site_tests/graphics_Piglit/graphics_Piglit.py:8: import logging, os, re,
shutil
On 2011/03/26 00:58:57, dalec wrote:
> Unused shutil.
It is used now to copy a file.
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
client/site_tests/graphics_Piglit/graphics_Piglit.py:13: class
graphics_Piglit(cros_ui_test.UITest):
Will keep UItest for now.
On 2011/03/26 00:58:57, dalec wrote:
> You should be fine with test.test here, you're not using any of the UITest
> functionality.
>
> From the logs you linked before I saw a bunch of crashes being detected
though,
> so maybe it's a good idea to keep.
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
client/site_tests/graphics_Piglit/graphics_Piglit.py:19: #if not
os.path.exists(dst_path):
On 2011/03/26 00:58:57, dalec wrote:
> If you go the ebuild route, the tarball should be copied into the
> graphics_Piglit directory, not /usr/local/autotest/packages.
>
> That way you don't need to load SYSROOT or anything which may potentially
break
> on the Autotest server in the lab.
I've decided to not use ebuild and follow the suggested route of checking a
tarball in. The problem is that piglit needs glut which needs to be created by
ebuild and is indexed by SYSROOT. How do I avoid this?
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
client/site_tests/graphics_Piglit/graphics_Piglit.py:27: results_path =
os.path.join(self.bindir,
On 2011/03/26 00:58:57, dalec wrote:
> self.outputdir + '/cros-driver' should work.
>
> There's also self.resultsdir. You can find a full list at
> client/common_lib/test.py
Done.
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
File client/site_tests/graphics_Piglit/src/KeepEmptySrcDirectory.txt (right):
http://codereview.chromium.org/6745001/diff/1001/client/site_tests/graphics_P...
client/site_tests/graphics_Piglit/src/KeepEmptySrcDirectory.txt:1: It is needed
for Autotest which stores .version information.
On 2011/03/26 00:58:57, dalec wrote:
> You could instead add the following to the setup clause:
>
> if not os.path.exists(self.srcdir):
> os.mkdir(self.srcdir)
Done.
DaleCurtis
+zdenek As with my comment on your other CL, if this breaks on ARM it'll ...
9 years, 8 months ago
(2011-04-04 17:51:55 UTC)
#4
I am ready to check in. I did some more testing over the weekend and ...
9 years, 8 months ago
(2011-04-11 18:08:20 UTC)
#9
I am ready to check in. I did some more testing over the weekend and I did
notice that using test.test is not enough as nearly all graphics tests need
the UITest after all for the behind the scenes auto login feature. (The test
would run only when already logged in as guest.) I reverted this and it has
been running stable 20 times in a row. Otherwise I decided to strip symbols
from the binaries, as this reduced transmission times and saves about 5
minutes of testing time per run.
What do I need to do to now to make this go life? We need
- the lab machines to boot before/after the test runs
- plot count_subtests_failTotal unique crashes:
count_subtests_pass
Total unique crashes:
- ideally
link
/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html
to the results pages as this contains detailed results for each run
I can file a bug if needed.
On Fri, Apr 8, 2011 at 2:50 PM, Ilja Friedel <ihf@google.com> wrote:
> I spoke with Zel and Monday is good. Thanks!
>
> On Fri, Apr 8, 2011 at 2:47 PM, <dalecurtis@chromium.org> wrote:
>
>> LGTM, but unless you're going to be around late you might want to wait
>> until
>> monday morning to check in, since it could break the build.
>>
>>
>> On 2011/04/08 21:42:24, ihf wrote:
>>
>>> Addressing all feedback by dalec.
>>>
>>
>>
>>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/piglit.py
>>> File client/deps/piglit/piglit.py (right):
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/piglit.p...
>>
>>> client/deps/piglit/piglit.py:7: import os, shutil, re
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > alphabetize imports.
>>>
>>
>> Done.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/piglit.p...
>>
>>> client/deps/piglit/piglit.py:7: import os, shutil, re
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > alphabetize imports.
>>>
>>
>> Done.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/piglit.p...
>>
>>> client/deps/piglit/piglit.py:14: # IHF: piglit only builds on x86 right
>>>
>> (Tegra2
>>
>>> only supporting GLES)
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > # TODO(ihf): ...
>>>
>>
>> Done.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/piglit.p...
>>
>>> client/deps/piglit/piglit.py:36: print 'WARNING: Skipping piglit build.
>>> piglit
>>> needs OpenGL (x86 boards)'
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Logging.
>>>
>>
>> Done.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/src/READ...
>>
>>> File client/deps/piglit/src/README.txt (right):
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/src/READ...
>>
>>> client/deps/piglit/src/README.txt:7:
>>> third_party/chromiumos-overlay/chromeos-base/autotest-deps-piglit/
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Update text.
>>>
>>
>> Done.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/src/cros...
>>
>>> File client/deps/piglit/src/cros-driver.tests (right):
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/deps/piglit/src/cros...
>>
>>> client/deps/piglit/src/cros-driver.tests:2: # -*- coding: utf-8 -*-
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Non-standard header, use the Google Chromium one.
>>>
>>
>> I've updated the file. There is no need to review it though (especially
>>> for
>>> style) as it is part of Piglit.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/site_tests/graphics_...
>>
>>> File client/site_tests/graphics_Piglit/control (right):
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/site_tests/graphics_...
>>
>>> client/site_tests/graphics_Piglit/control:9: !!! To be decided later when
>>> the
>>> suite passes or fails, as this might require
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Use "TODO(ihf):" as the format.
>>>
>>
>> Adjusted comment.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/site_tests/graphics_...
>>
>>> File client/site_tests/graphics_Piglit/graphics_Piglit.py (right):
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/site_tests/graphics_...
>>
>>> client/site_tests/graphics_Piglit/graphics_Piglit.py:20: # hard wiring
>>> the
>>> cros-driver.test config file for now
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Comment still valid?
>>>
>>
>> modified comment
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/site_tests/graphics_...
>>
>>> client/site_tests/graphics_Piglit/graphics_Piglit.py:47: return
>>> error.TestFail('test runs only on x86')
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Should be error.TestError.
>>>
>>
>> Done.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/site_tests/graphics_...
>>
>>> client/site_tests/graphics_Piglit/graphics_Piglit.py:47: return
>>> error.TestFail('test runs only on x86')
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Should be error.TestError.
>>>
>>
>> Done.
>>>
>>
>>
>>
>>
http://codereview.chromium.org/6745001/diff/14001/client/site_tests/graphics_...
>>
>>> client/site_tests/graphics_Piglit/graphics_Piglit.py:49: # get passed
>>> On 2011/04/08 16:58:18, dalec wrote:
>>> > Could do all three of these sections together plus the logging and
>>> keyvals
>>> below
>>> > with a for loop over the array ['pass', 'fail', 'warn', 'skip']
>>>
>>
>> You are right. But what I am doing here is parsing a piglit internal
>>> result
>>>
>> file
>>
>>> for tokens and translating them to tokens for autotest. That I chose to
>>> name
>>> them the same makes the code look simple. But if piglit ever changed its
>>>
>> output
>>
>>> the for-loop would not work anymore. This way it is IMHO easier to
>>> maintain.
>>>
>>
>>
>>
>> http://codereview.chromium.org/6745001/
>>
>
>
DaleCurtis
Since you've switched back to UITest, you'll need to blacklist the crashing processes to prevent ...
9 years, 8 months ago
(2011-04-11 18:34:54 UTC)
#10
Since you've switched back to UITest, you'll need to blacklist the crashing
processes to prevent them from causing regression failures.
If you file an issue against truty he can help you get the plots set up. The
change will go live as soon as this is checked in and will run against the next
build with the regression suite.
We don't have a way to link the report in just yet. Since there are no failure
conditions just yet, it's more for informational purposes anyways. I assume you
will add a failure condition in the future, when that happens we can think about
how to include the report with emails.
On 2011/04/11 18:08:20, ihf_google.com wrote:
> I am ready to check in. I did some more testing over the weekend and I did
> notice that using test.test is not enough as nearly all graphics tests need
> the UITest after all for the behind the scenes auto login feature. (The test
> would run only when already logged in as guest.) I reverted this and it has
> been running stable 20 times in a row. Otherwise I decided to strip symbols
> from the binaries, as this reduced transmission times and saves about 5
> minutes of testing time per run.
>
> What do I need to do to now to make this go life? We need
> - the lab machines to boot before/after the test runs
> - plot count_subtests_failTotal unique crashes:
> count_subtests_pass
> Total unique crashes:
> - ideally
> link
>
/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html
> to the results pages as this contains detailed results for each run
>
> I can file a bug if needed.
>
ihf
1) Added blacklist. Notice that list of unique crashes dropped to 5 from 10. The ...
9 years, 8 months ago
(2011-04-11 20:20:01 UTC)
#11
1) Added blacklist. Notice that list of unique crashes dropped to 5 from 10.
The remaining crashes *are* in the blacklist. Not sure this is expected.
Crashes detected during testing:
-------------------------------------
fbo-depth-sample-compare sig 11
graphics_Piglit/graphics_Piglit
getuniform-01 sig 11
graphics_Piglit/graphics_Piglit
glsl-bug-22603 sig 11
graphics_Piglit/graphics_Piglit
glsl-fs-color-matrix sig 11
graphics_Piglit/graphics_Piglit
glsl-fs-discard-02 sig 11
graphics_Piglit/graphics_Piglit
-------------------------------------
Total unique crashes: 5
2) Filed bug for truty
http://code.google.com/p/chromium-os/issues/detail?id=14043
<http://code.google.com/p/chromium-os/issues/detail?id=14043>3) Well. "Just
for informational purposes" is why we are running any test.
The question is how the results are interpreted and who responds to problems
with
the information provided. Piglit has a command to produce output webpages to
compare
N different result. This could be either N different boards or comparing
against last (checkin,
day, weekly, monthly) runs. I understand this is functionality that you are
trying to offer in autotest, but as it is already built into piglit,
lightweight and solved
well I would like to have access to this information in the long run.
For now I am fine without a link as long as one can find the html page
manually
from the results on disk (which I think is true).
On Mon, Apr 11, 2011 at 11:34 AM, <dalecurtis@chromium.org> wrote:
> Since you've switched back to UITest, you'll need to blacklist the crashing
> processes to prevent them from causing regression failures.
>
> If you file an issue against truty he can help you get the plots set up.
> The
> change will go live as soon as this is checked in and will run against the
> next
> build with the regression suite.
>
> We don't have a way to link the report in just yet. Since there are no
> failure
> conditions just yet, it's more for informational purposes anyways. I assume
> you
> will add a failure condition in the future, when that happens we can think
> about
> how to include the report with emails.
>
>
> On 2011/04/11 18:08:20, ihf_google.com wrote:
>
>> I am ready to check in. I did some more testing over the weekend and I did
>> notice that using test.test is not enough as nearly all graphics tests
>> need
>> the UITest after all for the behind the scenes auto login feature. (The
>> test
>> would run only when already logged in as guest.) I reverted this and it
>> has
>> been running stable 20 times in a row. Otherwise I decided to strip
>> symbols
>> from the binaries, as this reduced transmission times and saves about 5
>> minutes of testing time per run.
>>
>
> What do I need to do to now to make this go life? We need
>> - the lab machines to boot before/after the test runs
>> - plot count_subtests_failTotal unique crashes:
>> count_subtests_pass
>> Total unique crashes:
>> - ideally
>> link
>>
>
>
>
/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html
>
>> to the results pages as this contains detailed results for each run
>>
>
> I can file a bug if needed.
>>
>
>
> http://codereview.chromium.org/6745001/
>
ihf
After talking to Dale added fix to regex in cros_ui_test.py and now all crashes are ...
9 years, 8 months ago
(2011-04-11 20:52:17 UTC)
#12
After talking to Dale added fix to regex in cros_ui_test.py and now all
crashes are blacklisted as intended.
On Mon, Apr 11, 2011 at 1:19 PM, Ilja Friedel <ihf@google.com> wrote:
> 1) Added blacklist. Notice that list of unique crashes dropped to 5 from
> 10. The remaining crashes *are* in the blacklist. Not sure this is expected.
> Crashes detected during testing:
> -------------------------------------
> fbo-depth-sample-compare sig 11
> graphics_Piglit/graphics_Piglit
> getuniform-01 sig 11
> graphics_Piglit/graphics_Piglit
> glsl-bug-22603 sig 11
> graphics_Piglit/graphics_Piglit
> glsl-fs-color-matrix sig 11
> graphics_Piglit/graphics_Piglit
> glsl-fs-discard-02 sig 11
> graphics_Piglit/graphics_Piglit
> -------------------------------------
> Total unique crashes: 5
>
> 2) Filed bug for truty
> http://code.google.com/p/chromium-os/issues/detail?id=14043
>
> <http://code.google.com/p/chromium-os/issues/detail?id=14043>3) Well.
> "Just for informational purposes" is why we are running any test.
> The question is how the results are interpreted and who responds to
> problems with
> the information provided. Piglit has a command to produce output webpages
> to compare
> N different result. This could be either N different boards or comparing
> against last (checkin,
> day, weekly, monthly) runs. I understand this is functionality that you
> are
> trying to offer in autotest, but as it is already built into piglit,
> lightweight and solved
> well I would like to have access to this information in the long run.
>
> For now I am fine without a link as long as one can find the html page
> manually
> from the results on disk (which I think is true).
>
>
> On Mon, Apr 11, 2011 at 11:34 AM, <dalecurtis@chromium.org> wrote:
>
>> Since you've switched back to UITest, you'll need to blacklist the
>> crashing
>> processes to prevent them from causing regression failures.
>>
>> If you file an issue against truty he can help you get the plots set up.
>> The
>> change will go live as soon as this is checked in and will run against the
>> next
>> build with the regression suite.
>>
>> We don't have a way to link the report in just yet. Since there are no
>> failure
>> conditions just yet, it's more for informational purposes anyways. I
>> assume you
>> will add a failure condition in the future, when that happens we can think
>> about
>> how to include the report with emails.
>>
>>
>> On 2011/04/11 18:08:20, ihf_google.com wrote:
>>
>>> I am ready to check in. I did some more testing over the weekend and I
>>> did
>>> notice that using test.test is not enough as nearly all graphics tests
>>> need
>>> the UITest after all for the behind the scenes auto login feature. (The
>>> test
>>> would run only when already logged in as guest.) I reverted this and it
>>> has
>>> been running stable 20 times in a row. Otherwise I decided to strip
>>> symbols
>>> from the binaries, as this reduced transmission times and saves about 5
>>> minutes of testing time per run.
>>>
>>
>> What do I need to do to now to make this go life? We need
>>> - the lab machines to boot before/after the test runs
>>> - plot count_subtests_failTotal unique crashes:
>>> count_subtests_pass
>>> Total unique crashes:
>>> - ideally
>>> link
>>>
>>
>>
>>
/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html
>>
>>> to the results pages as this contains detailed results for each run
>>>
>>
>> I can file a bug if needed.
>>>
>>
>>
>> http://codereview.chromium.org/6745001/
>>
>
>
Issue 6745001: Add piglit autotest
(Closed)
Created 9 years, 9 months ago by ilja
Modified 9 years, 7 months ago
Reviewers: marcheu, petkov, DaleCurtis, ihf
Base URL: ssh://gitrw.chromium.org:9222/autotest.git@master
Comments: 37
Chromium Code Reviews
Issue
6745001:
Add piglit autotest (Closed)
