Add folders for skimage expectations, plus one set

Add folders for skimage expectations, plus one set

Expectations json files will be stored here for running skimage.

Once we move to git, this directory will be at a higher level, but
keeping here for now for simplicity.

(SkipBuildbotRuns)

BUG=https://code.google.com/p/skia/issues/detail?id=1241
R=borenet@google.com

Committed: https://code.google.com/p/skia/source/detail?r=9758

[scroggo, 2013-06-25 20:21:30]

This CL uses the directory structure I discussed with Elliot:

expectations
 - skimage
   - base-linux
   - ...
 - gm
   - base-linux
   - ...

To make sure skimage and gm stay in sync, would it make more sense to have the
following?

expectations
 - base-linux
   - skimage
   - gm

Or, since I expect there will only be one expectation file per subdir, would a
flat directory make sense?

expectations
 - gm-base-linux.json
 - skimage-base-linux.json
 - ...

[borenet, 2013-06-25 20:31:54]

On 2013/06/25 20:21:30, scroggo wrote:
> This CL uses the directory structure I discussed with Elliot:
> 
> expectations
>  - skimage
>    - base-linux
>    - ...
>  - gm
>    - base-linux
>    - ...
> 
> To make sure skimage and gm stay in sync, would it make more sense to have the
> following?
> 
> expectations
>  - base-linux
>    - skimage
>    - gm
> 
> Or, since I expect there will only be one expectation file per subdir, would a
> flat directory make sense?
> 
> expectations
>  - gm-base-linux.json
>  - skimage-base-linux.json
>  - ...

Personally, I like the flat directory, or maybe a flat directory for each type
of expectation:
expectations
 - skimage
    - base-linux
    - ...
 - gm
    - base-linux
    - ...

I still wonder whether we need names like "base-*" any more.  They make
rebaselining harder, because there isn't a 1:1 mapping from expectation dir to
builder name.  The only downside I can think of is that we couldn't "share"
expectations between builders, but since we'll only be storing checksums and
text files, I don't think it's much worse.  Maybe this is worth a larger
discussion.

[scroggo, 2013-06-25 20:55:03]

On 2013/06/25 20:31:54, borenet wrote:
> On 2013/06/25 20:21:30, scroggo wrote:
> > This CL uses the directory structure I discussed with Elliot:
> > 
> > expectations
> >  - skimage
> >    - base-linux
> >    - ...
> >  - gm
> >    - base-linux
> >    - ...
> > 
> > To make sure skimage and gm stay in sync, would it make more sense to have
the
> > following?
> > 
> > expectations
> >  - base-linux
> >    - skimage
> >    - gm
> > 
> > Or, since I expect there will only be one expectation file per subdir, would
a
> > flat directory make sense?
> > 
> > expectations
> >  - gm-base-linux.json
> >  - skimage-base-linux.json
> >  - ...
> 
> Personally, I like the flat directory, or maybe a flat directory for each type
> of expectation:
> expectations
>  - skimage
>     - base-linux
>     - ...
>  - gm
>     - base-linux
>     - ...
> 
> I still wonder whether we need names like "base-*" any more.  They make
> rebaselining harder, because there isn't a 1:1 mapping from expectation dir to
> builder name.  The only downside I can think of is that we couldn't "share"
> expectations between builders, but since we'll only be storing checksums and
> text files, I don't think it's much worse.  Maybe this is worth a larger
> discussion.

Latest CL uses a flat structure. I've added SkipBuildBotRuns to the description
since nothing refers to these new files.

Current plan is to check in so I can continue working on
https://codereview.chromium.org/16226005/ and have the larger discussion next
week once Elliot returns.

[borenet, 2013-06-25 20:59:00]

LGTM with question.

https://codereview.chromium.org/17737002/diff/4001/expectations/skimage/base-...
File expectations/skimage/base-shuttle_ubuntu12_ati5770-actual.json (right):

https://codereview.chromium.org/17737002/diff/4001/expectations/skimage/base-...
expectations/skimage/base-shuttle_ubuntu12_ati5770-actual.json:1: {
Is this file intentionally called "*-actual.json"?  "*-expected.json" seems more
appropriate, or since we're already inside an expectations dir, maybe just
"base-shuttle_ubuntu12_ati5770.json"?

[scroggo, 2013-06-25 21:03:45]

https://codereview.chromium.org/17737002/diff/4001/expectations/skimage/base-...
File expectations/skimage/base-shuttle_ubuntu12_ati5770-actual.json (right):

https://codereview.chromium.org/17737002/diff/4001/expectations/skimage/base-...
expectations/skimage/base-shuttle_ubuntu12_ati5770-actual.json:1: {
On 2013/06/25 20:59:00, borenet wrote:
> Is this file intentionally called "*-actual.json"?  "*-expected.json" seems
more
> appropriate, or since we're already inside an expectations dir, maybe just
> "base-shuttle_ubuntu12_ati5770.json"?

D'oh! No, I meant -expected.json, but that is redundant.

[scroggo, 2013-06-25 21:04:06]

Committed patchset #3 manually as r9758 (presubmit successful).

[epoger, 2013-07-08 16:12:28]

LGTM

> I still wonder whether we need names like "base-*" any more.  They make
> rebaselining harder, because there isn't a 1:1 mapping from expectation dir to
> builder name.  The only downside I can think of is that we couldn't "share"
> expectations between builders, but since we'll only be storing checksums and
> text files, I don't think it's much worse.  Maybe this is worth a larger
> discussion.

Yeah, I think a structure like this probably makes the most sense:

expectations
  gm
    Ubuntu12-ShuttleA-ATI5770-x86-Debug.json
    Ubuntu12-ShuttleA-ATI5770-x86-Release.json
    Ubuntu12-ShuttleA-ATI5770-x86_64-Debug.json
    Ubuntu12-ShuttleA-ATI5770-x86_64-Release.json
  skimage
    Ubuntu12-ShuttleA-ATI5770-x86-Debug.json
    Ubuntu12-ShuttleA-ATI5770-x86-Release.json
    Ubuntu12-ShuttleA-ATI5770-x86_64-Debug.json
    Ubuntu12-ShuttleA-ATI5770-x86_64-Release.json

[scroggo, 2013-07-09 01:15:20]

On 2013/07/08 16:12:28, epoger wrote:
> LGTM
> 
> > I still wonder whether we need names like "base-*" any more.  They make
> > rebaselining harder, because there isn't a 1:1 mapping from expectation dir
to
> > builder name.  The only downside I can think of is that we couldn't "share"
> > expectations between builders, but since we'll only be storing checksums and
> > text files, I don't think it's much worse.  Maybe this is worth a larger
> > discussion.
> 
> Yeah, I think a structure like this probably makes the most sense:
> 
> expectations
>   gm
>     Ubuntu12-ShuttleA-ATI5770-x86-Debug.json
>     Ubuntu12-ShuttleA-ATI5770-x86-Release.json
>     Ubuntu12-ShuttleA-ATI5770-x86_64-Debug.json
>     Ubuntu12-ShuttleA-ATI5770-x86_64-Release.json
>   skimage
>     Ubuntu12-ShuttleA-ATI5770-x86-Debug.json
>     Ubuntu12-ShuttleA-ATI5770-x86-Release.json
>     Ubuntu12-ShuttleA-ATI5770-x86_64-Debug.json
>     Ubuntu12-ShuttleA-ATI5770-x86_64-Release.json

So in other words, I should just use the builder_name? This would solve
https://code.google.com/p/skia/issues/detail?id=1399

[epoger, 2013-07-12 15:11:15]

On 2013/07/09 01:15:20, scroggo wrote:
> So in other words, I should just use the builder_name? This would solve
> https://code.google.com/p/skia/issues/detail?id=1399

Yes, I think so.  I intend to move GM in that direction also.