gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout - Issue 12825005: gm self-test: make sure we report failures in individual rendering modes

Side by Side Diff: gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout

Issue 12825005: gm self-test: make sure we report failures in individual rendering modes (Closed) Base URL: http://skia.googlecode.com/svn/trunk/

Patch Set: update gm self-test expectations to match patchset 3 Created 7 years, 9 months ago

Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.

Jump to:

View unified diff | Download patch | Annotate | Revision Log

« gm/gmmain.cpp ('K') | « gm/gmmain.cpp ('k') | gm/tests/outputs/compared-against-different-pixels-json/output-expected/stdout » ('j') | gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt » ('J')
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Hide Comments ('s')

OLD	NEW
1 GM: reading from gm/tests/inputs/images/different-pixels	1 GM: reading from gm/tests/inputs/images/different-pixels

2 GM: drawing... selftest1 [300 200]	2 GM: drawing... selftest1 [300 200]

3 GM: Ran 1 tests: 0 passed, 1 failed, 0 missing reference images	3 GM: Ran 1 tests: 0 passed, 1 failed, 0 with drawing mode discrepancies, 0 missin g reference images
	epoger 2013/03/14 22:36:27 we now report the number of drawing mode discrepan we now report the number of drawing mode discrepancies for every run borenet 2013/03/15 20:15:53 I like this, but I might rather the wording change Show quoted text On 2013/03/14 22:36:27, epoger wrote: > we now report the number of drawing mode discrepancies for every run I like this, but I might rather the wording change to something like, "%d passed, %d did not match expectations, %d had differences between drawing modes, %d missing reference images" Just to make it clear what we now mean by "failed." epoger 2013/03/15 22:05:55 One thing you should be aware of is that a single Show quoted text On 2013/03/15 20:15:53, borenet wrote: > On 2013/03/14 22:36:27, epoger wrote: > > we now report the number of drawing mode discrepancies for every run > > I like this, but I might rather the wording change to something like, "%d > passed, %d did not match expectations, %d had differences between drawing modes, > %d missing reference images" Just to make it clear what we now mean by > "failed." One thing you should be aware of is that a single test can fail in more than one way. For example, a test could have differences across drawing modes and ALSO be missing expectations. So this could get confusing. ("Hey, it says 1 test had failure A, and 1 test had failure B, but only 1 test ran!") I don't know the best way to communicate that. Any ideas? borenet 2013/03/18 12:26:37 Is it critical that the summary be one line? What Show quoted text On 2013/03/15 22:05:55, epoger wrote: > On 2013/03/15 20:15:53, borenet wrote: > > On 2013/03/14 22:36:27, epoger wrote: > > > we now report the number of drawing mode discrepancies for every run > > > > I like this, but I might rather the wording change to something like, "%d > > passed, %d did not match expectations, %d had differences between drawing > modes, > > %d missing reference images" Just to make it clear what we now mean by > > "failed." > > One thing you should be aware of is that a single test can fail in more than one > way. For example, a test could have differences across drawing modes and ALSO > be missing expectations. So this could get confusing. ("Hey, it says 1 test > had failure A, and 1 test had failure B, but only 1 test ran!") > > I don't know the best way to communicate that. Any ideas? Is it critical that the summary be one line? What if we listed failures by gm: selftest1: did not match expecations othergm02: missing reference image othergm03: drawing mode discrepancy, missing reference image othergm04: drawing mode discrepancy (can we be more specific?) The downside here is that we'd get much more output, especially when we don't provide any expecations, but I can't think of a scenario where we wouldn't want to know exactly which gms had failures. As for the missing expectations, I guess we could summarize those in a single line, although it's probably also helpful to know which are missing expecations so that we can provide expectations. epoger 2013/03/21 16:08:32 How about if gm generates something similar to the Show quoted text On 2013/03/18 12:26:37, borenet wrote: > Is it critical that the summary be one line? What if we listed failures by gm: > > selftest1: did not match expecations > othergm02: missing reference image > othergm03: drawing mode discrepancy, missing reference image > othergm04: drawing mode discrepancy (can we be more specific?) > > The downside here is that we'd get much more output, especially when we don't > provide any expecations, but I can't think of a scenario where we wouldn't want > to know exactly which gms had failures. As for the missing expectations, I > guess we could summarize those in a single line, although it's probably also > helpful to know which are missing expecations so that we can provide > expectations. How about if gm generates something similar to the skdiff output (e.g. https://code.google.com/p/skia/source/browse/trunk/tools/tests/skdiff/test2/o... ), but with the addition of one single-line summary at the top? (That single-line summary is probably handy in many cases, and cheap to generate.) So, gm/tests/outputs/pipe-playback-failure/output-expected/stdout would look like this: GM: summary: ran 1 tests: 0 NoGpuContext, 1 MismatchRenderingMode, 0 MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage GM: [] 0 NoGpuContext : GM: [] 1 MismatchRenderingMode : comparison/selftest1-pipe GM: [] 0 MismatchExpectations : GM: [_] 0 MissingExpectations : GM: [] 0 ErrorWritingImage : GM: (results marked with [] will cause nonzero return value) (I have also changed the categories a bit, making MismatchRenderingMode distinct from MismatchExpectations.) And gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout would look like this: GM: summary: ran 1 tests: 0 NoGpuContext, 0 MismatchRenderingMode, 2 MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage GM: [] 0 NoGpuContext : GM: [] 0 MismatchRenderingMode : GM: [] 2 MismatchExpectations : 8888/selftest1 565/selftest1 GM: [_] 0 MissingExpectations : GM: [] 0 ErrorWritingImage : GM: (results marked with [] will cause nonzero return value) One thing you will note in this latter case is that just 1 test was run, but 2 errors were reported! That's because the test was run in 2 different config modes. Do you think that would be problematic? I think we could find ways to improve that part of the reporting over time... borenet 2013/03/21 18:12:47 Consistency is good! I like it, but as you demons Show quoted text On 2013/03/21 16:08:32, epoger wrote: > On 2013/03/18 12:26:37, borenet wrote: > > Is it critical that the summary be one line? What if we listed failures by > gm: > > > > selftest1: did not match expecations > > othergm02: missing reference image > > othergm03: drawing mode discrepancy, missing reference image > > othergm04: drawing mode discrepancy (can we be more specific?) > > > > The downside here is that we'd get much more output, especially when we don't > > provide any expecations, but I can't think of a scenario where we wouldn't > want > > to know exactly which gms had failures. As for the missing expectations, I > > guess we could summarize those in a single line, although it's probably also > > helpful to know which are missing expecations so that we can provide > > expectations. > > How about if gm generates something similar to the skdiff output (e.g. > https://code.google.com/p/skia/source/browse/trunk/tools/tests/skdiff/test2/o... > ), but with the addition of one single-line summary at the top? (That > single-line summary is probably handy in many cases, and cheap to generate.) > > So, gm/tests/outputs/pipe-playback-failure/output-expected/stdout would look > like this: > > GM: summary: ran 1 tests: 0 NoGpuContext, 1 MismatchRenderingMode, 0 > MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage > GM: [] 0 NoGpuContext : > GM: [] 1 MismatchRenderingMode : comparison/selftest1-pipe > GM: [] 0 MismatchExpectations : > GM: [_] 0 MissingExpectations : > GM: [] 0 ErrorWritingImage : > GM: (results marked with [] will cause nonzero return value) > > (I have also changed the categories a bit, making MismatchRenderingMode distinct > from MismatchExpectations.) > > And > gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout > would look like this: > > GM: summary: ran 1 tests: 0 NoGpuContext, 0 MismatchRenderingMode, 2 > MismatchExpectations, 0 MissingExpectations, 0 ErrorWritingImage > GM: [] 0 NoGpuContext : > GM: [] 0 MismatchRenderingMode : > GM: [] 2 MismatchExpectations : 8888/selftest1 565/selftest1 > GM: [_] 0 MissingExpectations : > GM: [] 0 ErrorWritingImage : > GM: (results marked with [] will cause nonzero return value) > > One thing you will note in this latter case is that just 1 test was run, but 2 > errors were reported! That's because the test was run in 2 different config > modes. Do you think that would be problematic? I think we could find ways to > improve that part of the reporting over time... Consistency is good! I like it, but as you demonstrated, it doesn't solve the issue of a single GM failing in multiple ways. However, I don't really think that's a big problem, since we rarely (to my knowledge) use the number of failures as opposed to wanting to know what failed and why.
OLD	NEW