Add AutoIndex

Add AutoIndex to datastore Testable interface.

This allows test writers to set Consistency(true) and AutoIndex(true), and pretend like the datastore is a fully consistent database which always has perfect indexes. Will be used for the recursive transaction manager to maintained tiered automatic mini-datastores in memory.

Also add some more tests for queries in transactions, and YAML rendering for
*IndexDefinition.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org, tandrii@chromium.org, vadimsh@chromium.org
BUG=533021
, https://github.com/luci/gae/issues/9

Committed: https://github.com/luci/gae/commit/ebfadf7ba75ef6e061d957f906b9370d58a498a7

[Vadim Sh., 2015-09-24 19:17:30]

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go
File service/datastore/index.go (right):

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:240: ret := &bytes.Buffer{}
nit: just bytes.Buffer{}, no need for dynamic allocation

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:251: if err != nil {
just remove errcheck from pre-commit-go.yml >_< Checking Fprintf errors is
clearly paranoia.

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:265: indent++
:) just pass indent explicitly, e.g.

ws("properties:", 1)
...
ws("direction: desc", 2)

https://codereview.chromium.org/1364333002/diff/1/service/datastore/testable.go
File service/datastore/testable.go (right):

https://codereview.chromium.org/1364333002/diff/1/service/datastore/testable....
service/datastore/testable.go:52: // AutoIndex controls the index creation
behavior. If it is set to true, then
It probably doesn't make sense without Consistent(true)? Or new index is
immediately usable without a call to CatchupIndexes?

[iannucci, 2015-09-24 19:45:00]

On 2015/09/24 at 19:17:30, vadimsh wrote:
> https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go
> File service/datastore/index.go (right):
> 
>
https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
> service/datastore/index.go:240: ret := &bytes.Buffer{}
> nit: just bytes.Buffer{}, no need for dynamic allocation
> 
>
https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
> service/datastore/index.go:251: if err != nil {
> just remove errcheck from pre-commit-go.yml >_< Checking Fprintf errors is
clearly paranoia.
> 
>
https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
> service/datastore/index.go:265: indent++
> :) just pass indent explicitly, e.g.
> 
> ws("properties:", 1)
> ...
> ws("direction: desc", 2)
> 
>
https://codereview.chromium.org/1364333002/diff/1/service/datastore/testable.go
> File service/datastore/testable.go (right):
> 
>
https://codereview.chromium.org/1364333002/diff/1/service/datastore/testable....
> service/datastore/testable.go:52: // AutoIndex controls the index creation
behavior. If it is set to true, then
> It probably doesn't make sense without Consistent(true)? Or new index is
immediately usable without a call to CatchupIndexes?

It makes sense if you are making an ancestor query, for example.

[Vadim Sh., 2015-09-24 19:49:59]

lgtm (with nits in previous comment)

[iannucci, 2015-09-24 19:52:35]

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go
File service/datastore/index.go (right):

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:240: ret := &bytes.Buffer{}
On 2015/09/24 at 19:17:29, Vadim Sh. wrote:
> nit: just bytes.Buffer{}, no need for dynamic allocation

Common misconception: there's no actual difference. The compiler reserves the
right to allocate any 'stack' allocations on the heap, or vice versa.

https://golang.org/doc/faq#stack_or_heap

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:251: if err != nil {
On 2015/09/24 at 19:17:29, Vadim Sh. wrote:
> just remove errcheck from pre-commit-go.yml >_< Checking Fprintf errors is
clearly paranoia.

lol ok

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:265: indent++
On 2015/09/24 at 19:17:29, Vadim Sh. wrote:
> :) just pass indent explicitly, e.g.
> 
> ws("properties:", 1)
> ...
> ws("direction: desc", 2)

done :)

[Vadim Sh., 2015-09-24 19:57:50]

lgtm

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go
File service/datastore/index.go (right):

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:240: ret := &bytes.Buffer{}
On 2015/09/24 19:52:34, iannucci wrote:
> On 2015/09/24 at 19:17:29, Vadim Sh. wrote:
> > nit: just bytes.Buffer{}, no need for dynamic allocation
> 
> Common misconception: there's no actual difference. The compiler reserves the
> right to allocate any 'stack' allocations on the heap, or vice versa.
> 
> https://golang.org/doc/faq#stack_or_heap

"""In the current compilers, if a variable has its address taken, that variable
is a candidate for allocation on the heap. However, a basic escape analysis
recognizes some cases when such variables will not live past the return from the
function and can reside on the stack."""

I try to help escape analysis whenever I can :)

Anyhow, in this particular case "&" is just unnecessary symbol. But I don't care
much.

[iannucci, 2015-09-24 20:12:56]

The CQ bit was checked by iannucci@chromium.org

[iannucci, 2015-09-24 20:12:56]

The patchset sent to the CQ was uploaded after l-g-t-m from vadimsh@chromium.org
Link to the patchset: https://codereview.chromium.org/1364333002/#ps40001
(title: "symbollls")

[iannucci, 2015-09-24 20:12:59]

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go
File service/datastore/index.go (right):

https://codereview.chromium.org/1364333002/diff/1/service/datastore/index.go#...
service/datastore/index.go:240: ret := &bytes.Buffer{}
On 2015/09/24 at 19:57:50, Vadim Sh. wrote:
> On 2015/09/24 19:52:34, iannucci wrote:
> > On 2015/09/24 at 19:17:29, Vadim Sh. wrote:
> > > nit: just bytes.Buffer{}, no need for dynamic allocation
> > 
> > Common misconception: there's no actual difference. The compiler reserves
the
> > right to allocate any 'stack' allocations on the heap, or vice versa.
> > 
> > https://golang.org/doc/faq#stack_or_heap
> 
> """In the current compilers, if a variable has its address taken, that
variable is a candidate for allocation on the heap. However, a basic escape
analysis recognizes some cases when such variables will not live past the return
from the function and can reside on the stack."""
> 
> I try to help escape analysis whenever I can :)
> 
> Anyhow, in this particular case "&" is just unnecessary symbol. But I don't
care much.

it made more sense before I had the ws function. there were
fmt.Fprintf(&ret,...) everywhere.

[commit-bot: I haz the power, 2015-09-24 20:13:10]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1364333002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1364333002/40001

[commit-bot: I haz the power, 2015-09-24 20:14:24]

Committed patchset #3 (id:40001) as
https://github.com/luci/gae/commit/ebfadf7ba75ef6e061d957f906b9370d58a498a7