Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(259)

Issue 2819079: Make test failure messages more verbose (and thus useful to testing operators). (Closed)

Created:
10 years, 4 months ago by clchiou
Modified:
9 years, 7 months ago
CC:
chromium-os-reviews_chromium.org, sosa+cc_chromium.org, seano, ericli, petkov+cc_chromium.org
Base URL:
ssh://gitrw.chromium.org/autotest.git
Visibility:
Public.

Description

Patch Set 1 #

Patch Set 2 : More work on making TestFail message meaningful to factory QA #

Patch Set 3 : Make error criterion backward compatible #

Total comments: 25

Patch Set 4 : Put back English messages #

Patch Set 5 : Put back traditional Chinese messages #

Patch Set 6 : Rewrite some traditional Chinese messages #

Total comments: 23

Patch Set 7 : Add & rewrite messages #

Patch Set 8 : Revise external device removal logic #

Total comments: 1

Patch Set 9 : Add test fail message when dependent files missing #

Patch Set 10 : Add more test fail messages #

Unified diffs Side-by-side diffs Delta from patch set Stats (+41 lines, -9 lines) Patch
M client/site_tests/factory_DeveloperRecovery/factory_DeveloperRecovery.py View 2 3 4 5 6 7 8 1 chunk +4 lines, -1 line 0 comments Download
M client/site_tests/factory_ExtDisplay/factory_ExtDisplay.py View 3 chunks +10 lines, -2 lines 0 comments Download
M client/site_tests/factory_Keyboard/factory_Keyboard.py View 2 3 4 5 6 7 8 9 1 chunk +13 lines, -4 lines 0 comments Download
M client/site_tests/factory_Leds/factory_Leds.py View 2 3 4 5 6 7 8 9 1 chunk +8 lines, -1 line 0 comments Download
M client/site_tests/hardware_GPIOSwitches/hardware_GPIOSwitches.py View 2 chunks +6 lines, -1 line 0 comments Download

Messages

Total messages: 23 (0 generated)
clchiou
10 years, 4 months ago (2010-07-30 01:53:35 UTC) #1
clchiou
10 years, 4 months ago (2010-07-30 23:46:37 UTC) #2
clchiou
10 years, 4 months ago (2010-08-02 16:34:47 UTC) #3
Hung-Te
Hey, tammo - do you think we are really switching to zh-CN (Simplified Chinese)? Is ...
10 years, 4 months ago (2010-08-04 13:00:47 UTC) #4
clchiou
On 2010/08/04 13:00:47, Hung-Te wrote: > Hey, tammo - do you think we are really ...
10 years, 4 months ago (2010-08-04 18:14:00 UTC) #5
clchiou
Hi Hung-Te, I've addressed your comments. Thanks for reviewing. http://codereview.chromium.org/2819079/diff/7001/8001 File client/site_tests/factory_Camera/factory_Camera.py (right): http://codereview.chromium.org/2819079/diff/7001/8001#newcode191 client/site_tests/factory_Camera/factory_Camera.py:191: ...
10 years, 4 months ago (2010-08-04 20:27:42 UTC) #6
Hung-Te
> In fact, this issue is asking factory tests to > provide verbose messages when ...
10 years, 4 months ago (2010-08-05 04:00:34 UTC) #7
clchiou
Hi Hung-Te, Thanks for the timely review. I revert suite_Factory/control and rewrite some of messages ...
10 years, 4 months ago (2010-08-05 21:37:47 UTC) #8
Tammo Spalink
it looks like all the simplified chinese is back to traditional, right? we can tackle ...
10 years, 4 months ago (2010-08-06 04:35:56 UTC) #9
Hung-Te
http://codereview.chromium.org/2819079/diff/16003/7005 File client/site_tests/factory_EnableWriteProtect/factory_EnableWriteProtect.py (right): http://codereview.chromium.org/2819079/diff/16003/7005#newcode54 client/site_tests/factory_EnableWriteProtect/factory_EnableWriteProtect.py:54: raise error.TestError('ERROR: cannot select target %s' % Since this ...
10 years, 4 months ago (2010-08-06 09:34:12 UTC) #10
clchiou
Thanks for Hung-Te's and Tammo's help on making messages more clearer. Any comments are welcome. ...
10 years, 4 months ago (2010-08-06 20:53:03 UTC) #11
Hung-Te
LGTM
10 years, 4 months ago (2010-08-09 02:50:09 UTC) #12
Tammo Spalink
lgtm
10 years, 4 months ago (2010-08-09 02:54:22 UTC) #13
Nick Sanders
http://codereview.chromium.org/2819079/diff/41001/42006 File client/site_tests/factory_Keyboard/factory_Keyboard.py (right): http://codereview.chromium.org/2819079/diff/41001/42006#newcode143 client/site_tests/factory_Keyboard/factory_Keyboard.py:143: kbd_image = cairo.ImageSurface.create_from_png('%s.png' % layout) Can you put try/catch ...
10 years, 4 months ago (2010-08-10 07:32:35 UTC) #14
clchiou
Hi Nick, I grep'ed source code and found two places that depends on third-party files. ...
10 years, 4 months ago (2010-08-23 04:31:32 UTC) #15
Nick Sanders
Can you download the buildbot factory image, run it on an L13 and see in ...
10 years, 4 months ago (2010-08-23 05:38:37 UTC) #16
clchiou
Hi Nick, Many thanks for the comments and sorry for the late reply. Summary of ...
10 years, 3 months ago (2010-09-01 04:57:54 UTC) #17
Nick Sanders
Sounds good! On Tue, Aug 31, 2010 at 9:57 PM, <clchiou@google.com> wrote: > Hi Nick, ...
10 years, 3 months ago (2010-09-01 09:46:01 UTC) #18
Tom Wai-Hong Tam
On Wed, Sep 1, 2010 at 2:45 AM, Nick Sanders <nsanders@chromium.org> wrote: > Sounds good! ...
10 years, 3 months ago (2010-09-03 01:35:11 UTC) #19
clchiou
Hi Tom, Thanks for the suggestions. I was not proposing add try-catch block for these ...
10 years, 3 months ago (2010-09-03 03:54:47 UTC) #20
Tom Wai-Hong Tam
On Thu, Sep 2, 2010 at 8:54 PM, <clchiou@google.com> wrote: > Hi Tom, > > ...
10 years, 3 months ago (2010-09-03 04:51:26 UTC) #21
Tom Wai-Hong Tam
On Thu, Sep 2, 2010 at 8:54 PM, <clchiou@google.com> wrote: > Hi Tom, > > ...
10 years, 3 months ago (2010-09-03 04:51:35 UTC) #22
hungte
10 years, 3 months ago (2010-09-03 05:07:46 UTC) #23
On Fri, Sep 3, 2010 at 12:50 PM, Tom Wai-Hong Tam (談偉航) <
waihong@chromium.org> wrote:

> On Thu, Sep 2, 2010 at 8:54 PM, <clchiou@google.com> wrote:
>
>> Hi Tom,
>>
>> Thanks for the suggestions.  I was not proposing add try-catch block for
>> these
>> exceptions, but was uncertain what should be done.
>>
>> On the first suggestion, thanks for updating the status of GBB.  I think
>> it
>> would be okay I just leave it there until GBB is submitted, and then we
>> could
>> discuss what we should do.
>>
>> I just submitted a workable GBB. So you can try it now. Heard that there
> are 2 Mario prototypes in TW. It'd be very great if you can try the whole
> factory flow on it, including firmware update, matching HWID, and replacing
> the GBB.
>
>
>> On the second suggestion, I tested it again with a newer TOT factory image
>> including Hung-Te's patch.  It still failed as expected due to the legacy
>> BIOS
>> on L13, but I saw Hung-Te's patch was catching exceptions.
>
>

> Maybe you can check this issue with Hung-Te.
>
    You just get confused as I did :-)
    Well, Che-Liang simply wanted to say the patch worked as he expected,
    so there's no issue here.


>
>
> On 2010/09/03 01:35:11, Tom Wai-Hong Tam wrote:
>>
>>> On Wed, Sep 1, 2010 at 2:45 AM, Nick Sanders <mailto:
>>> nsanders@chromium.org>
>>>
>> wrote:
>>
>>  > Sounds good!
>>>
>>> >
>>> > On Tue, Aug 31, 2010 at 9:57 PM, <mailto:clchiou@google.com> wrote:
>>> >
>>> >> Hi Nick,
>>> >>
>>> >> Many thanks for the comments and sorry for the late reply.
>>> >>
>>> >> Summary of test result of buildbot TOT factory image on L13:
>>> >>
>>> >> 1 Tests with gpio setup failure
>>> >>  * factory_DeveloperRecovery and hardware_GPIOSwitches at runin test
>>> >>  * Command
>>> >>   /usr/sbin/gpio_setup
>>> >>  * AI: add try-catch block (done)
>>> >>
>>> >> 2 Tests with amixer command failed
>>> >>  * factory_ExtDisplay
>>> >>  * Command
>>> >>   amixer -c 0 cset name="IEC958 Playback Switch" on
>>> >>   amixer -c 0 cset name="IEC958 Playback Switch" off
>>> >>  * AI: add try-catch block (done)
>>> >>
>>> >> 3 Tests that have a bug
>>> >>  * factory_LightSensor
>>> >>  * bug: error.CmdError require 2 arguments but only 1 given
>>> >>  * AI: write to original author (done)
>>> >>
>>> >> 4 Test error that I am not sure whether it is supposed to happen
>>> >>  * factory_WriteGBB (at final stage1)
>>> >>   TestError: Unable to find GBB file: gbb_*
>>> >>  * hardware_Components and hardware_GPIOSwitches at runin test
>>> >>   TestError: Cannot get section [FV_BSTUB] from flashrom
>>> >>  * AI: ?
>>> >>
>>> >
>>> > For this one, as long as the test prints out these errors, it should be
>>> > fine.
>>> > The test should not pass,  however they should give good error
>>> messages.
>>> >
>>>
>>
>>  For the first item, since there is no gbb yet, so the failure is
>>> expected.
>>> We are working on it. Or you can grab one from current BIOS.
>>>
>>
>>  For the second iterm, the main problem of hardware_Components test failed
>>> is
>>> you run it on legacy bios. I agree should give good error messages, like
>>> catch the exception and raise another with better message. But I remember
>>> Hungte had a change to ignore errors and just return an empty string,
>>> please
>>> sync the code and try.
>>>
>>
>>
>>  >
>>> >
>>> >> As for the item 4, these tests are Google Required Tests for
>>> >> Google-blessed
>>> >> devices.  It is not my authority to decide these error should or
>>> should
>>> >> not
>>> >> happen in these tests.
>>> >>
>>> >
>>> > Since you are part of Google, it is actually your authority (since we
>>> are
>>> > the
>>> > ones blessing the devices). =)
>>> > But I agree with you that the errors reported are clear enough.
>>> >
>>> >
>>> >> * If they should happen, I think the original error message is clear
>>> >> enough; I
>>> >> could try to make it clearer, but I don't think there would be much I
>>> can
>>> >> do.
>>> >> * If they should not happen, we should file another bug for them.
>>> >>
>>> >> On 2010/08/23 05:38:37, Nick Sanders wrote:
>>> >>
>>> >>> Can you download the buildbot factory image, run it on an L13 and see
>>> in
>>> >>> which places if fails?
>>> >>> These two are good, but I think there are more places where it will
>>> fail,
>>> >>> but grep probably can''t find them, there will also be cases where
>>> the
>>> >>> code
>>> >>> assumes H2C, or is device specific, or whatever.
>>> >>>
>>> >>
>>> >>  Thanks,
>>> >>>   -Nick
>>> >>>
>>> >>
>>> >>  On Sun, Aug 22, 2010 at 9:31 PM, <mailto:clchiou@google.com> wrote:
>>> >>>
>>> >>
>>> >>  > Hi Nick,
>>> >>> >
>>> >>> > I grep'ed source code and found two places that depends on
>>> third-party
>>> >>> > files.
>>> >>> > They are in keyboard and leds testing.  If I missed any places,
>>> please
>>> >>> help
>>> >>> > me
>>> >>> > indicate where it is.  Thank you very much.
>>> >>> >
>>> >>> > On 2010/08/10 07:32:35, Nick Sanders wrote:
>>> >>> >
>>> >>> >> http://codereview.chromium.org/2819079/diff/41001/42006
>>> >>> >> File client/site_tests/factory_Keyboard/factory_Keyboard.py
>>> (right):
>>> >>> >>
>>> >>> >
>>> >>> >
>>> http://codereview.chromium.org/2819079/diff/41001/42006#newcode143
>>> >>> >> client/site_tests/factory_Keyboard/factory_Keyboard.py:143:
>>> kbd_image
>>> >>> =
>>> >>> >> cairo.ImageSurface.create_from_png('%s.png' % layout)
>>> >>> >> Can you put try/catch around this and add "keyboard img missing"
>>> and
>>> >>> >> "keyboard
>>> >>> >> binding missing" error msg
>>> >>> >>
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > http://codereview.chromium.org/2819079/show
>>> >>> >
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> http://codereview.chromium.org/2819079/show
>>> >>
>>> >
>>> >
>>>
>>
>>
>>
>>
>> http://codereview.chromium.org/2819079/show
>>
>
>

Powered by Google App Engine
This is Rietveld 408576698