gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout - Issue 12825005: gm self-test: make sure we report failures in individual rendering modes

Unified Diff: gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout

Issue 12825005: gm self-test: make sure we report failures in individual rendering modes (Closed) Base URL: http://skia.googlecode.com/svn/trunk/

Patch Set: update gm self-test expectations to match patchset 3 Created 7 years, 9 months ago

Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.

Jump to:

View side-by-side diff with in-line comments

Download patch

« gm/gmmain.cpp ('K') | « gm/gmmain.cpp ('k') | gm/tests/outputs/compared-against-different-pixels-json/output-expected/stdout » ('j') | gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt » ('J')
Expand Comments ('e') | Collapse Comments ('c') | Hide Comments ('s')

Index: gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout

===================================================================

--- gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout (revision 8160)

+++ gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout (working copy)

@@ -1,3 +1,3 @@

GM: reading from gm/tests/inputs/images/different-pixels

GM: drawing... selftest1 [300 200]

-GM: Ran 1 tests: 0 passed, 1 failed, 0 missing reference images

+GM: Ran 1 tests: 0 passed, 1 failed, 0 with drawing mode discrepancies, 0 missing reference images

epoger 2013/03/14 22:36:27 we now report the number of drawing mode discrepan

borenet 2013/03/15 20:15:53 I like this, but I might rather the wording change

epoger 2013/03/15 22:05:55 One thing you should be aware of is that a single

borenet 2013/03/18 12:26:37 Is it critical that the summary be one line? What

epoger 2013/03/21 16:08:32 How about if gm generates something similar to the

How about if gm generates something similar to the skdiff output (e.g. https://code.google.com/p/skia/source/browse/trunk/tools/tests/skdiff/test2/o... ), but with the addition of one single-line summary at the top? (That single-line summary is probably handy in many cases, and cheap to generate.) So, gm/tests/outputs/pipe-playback-failure/output-expected/stdout would look like this: GM: summary: ran 1 tests: 0 NoGpuContext, 1 MismatchRenderingMode, 0 MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage GM: [*] 0 NoGpuContext : GM: [*] 1 MismatchRenderingMode : comparison/selftest1-pipe GM: [*] 0 MismatchExpectations : GM: [_] 0 MissingExpectations : GM: [*] 0 ErrorWritingImage : GM: (results marked with [*] will cause nonzero return value) (I have also changed the categories a bit, making MismatchRenderingMode distinct from MismatchExpectations.) And gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout would look like this: GM: summary: ran 1 tests: 0 NoGpuContext, 0 MismatchRenderingMode, 2 MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage GM: [*] 0 NoGpuContext : GM: [*] 0 MismatchRenderingMode : GM: [*] 2 MismatchExpectations : 8888/selftest1 565/selftest1 GM: [_] 0 MissingExpectations : GM: [*] 0 ErrorWritingImage : GM: (results marked with [*] will cause nonzero return value) One thing you will note in this latter case is that just 1 test was run, but 2 errors were reported! That's because the test was run in 2 different config modes. Do you think that would be problematic? I think we could find ways to improve that part of the reporting over time...

borenet 2013/03/21 18:12:47 Consistency is good! I like it, but as you demons

On 2013/03/21 16:08:32, epoger wrote: > On 2013/03/18 12:26:37, borenet wrote: > > Is it critical that the summary be one line? What if we listed failures by > gm: > > > > selftest1: did not match expecations > > othergm02: missing reference image > > othergm03: drawing mode discrepancy, missing reference image > > othergm04: drawing mode discrepancy (can we be more specific?) > > > > The downside here is that we'd get much more output, especially when we don't > > provide any expecations, but I can't think of a scenario where we wouldn't > want > > to know exactly which gms had failures. As for the missing expectations, I > > guess we could summarize those in a single line, although it's probably also > > helpful to know which are missing expecations so that we can provide > > expectations. > > How about if gm generates something similar to the skdiff output (e.g. > https://code.google.com/p/skia/source/browse/trunk/tools/tests/skdiff/test2/o... > ), but with the addition of one single-line summary at the top? (That > single-line summary is probably handy in many cases, and cheap to generate.) > > So, gm/tests/outputs/pipe-playback-failure/output-expected/stdout would look > like this: > > GM: summary: ran 1 tests: 0 NoGpuContext, 1 MismatchRenderingMode, 0 > MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage > GM: [*] 0 NoGpuContext : > GM: [*] 1 MismatchRenderingMode : comparison/selftest1-pipe > GM: [*] 0 MismatchExpectations : > GM: [_] 0 MissingExpectations : > GM: [*] 0 ErrorWritingImage : > GM: (results marked with [*] will cause nonzero return value) > > (I have also changed the categories a bit, making MismatchRenderingMode distinct > from MismatchExpectations.) > > And > gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout > would look like this: > > GM: summary: ran 1 tests: 0 NoGpuContext, 0 MismatchRenderingMode, 2 > MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage > GM: [*] 0 NoGpuContext : > GM: [*] 0 MismatchRenderingMode : > GM: [*] 2 MismatchExpectations : 8888/selftest1 565/selftest1 > GM: [_] 0 MissingExpectations : > GM: [*] 0 ErrorWritingImage : > GM: (results marked with [*] will cause nonzero return value) > > One thing you will note in this latter case is that just 1 test was run, but 2 > errors were reported! That's because the test was run in 2 different config > modes. Do you think that would be problematic? I think we could find ways to > improve that part of the reporting over time...

Consistency is good! I like it, but as you demonstrated, it doesn't solve the issue of a single GM failing in multiple ways. However, I don't really think that's a big problem, since we rarely (to my knowledge) use the number of failures as opposed to wanting to know *what* failed and why.