Add filters for RawDatastore

Add filters for gae library.

Filters allow users to add their own wrappers around the underlying gae
implementation. They can do anything that an implementation could do, except
that their backend is another gae library interface, instead of the SDK or
memory implementations. Filters are also necessary (instead of just stacking
implementations with SetXXFactory functions), due to transactions, which change
the context object.

They will be used to implement (at least):
  * caching (hitting memcache before datastore)
  * transaction buffering (giving a self-consistent view of the datastore
    within a transaction, buffering all writes to the end of the transaction
    to make transaction sizes smaller, etc.)
  * featureBreaker (for tests, breaking certain APIs with certain errors)
  * counter (for tests, counting the number of times an API was hit)

It may also be used for:
  * ACL enforcement (rejecting access to datastore items the user can't get, 
    filtering queries, etc.)

Thi CL also implements a counter filter as a demo (which will also be used for
testing other filter implementations to ensure that the filter being tested).

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Committed: https://chromium.googlesource.com/infra/infra/+/ce7d584c71c3866963c8f12baa95032e092e2f3b

[Vadim Sh., 2015-07-08 14:15:39]

can you please explain what this CL does exactly and why you are doing it?

[iannucci, 2015-07-13 23:11:51]

On 2015/07/08 14:15:39, Vadim Sh. wrote:
> can you please explain what this CL does exactly and why you are doing it?

It adds the ability to modify the behavior of the various services (RDS, MC, TQ,
GI) by providing a proxy over their interface. It's necessary to implement
behavioral changes (like caching RDS access in memcache, buffering transactions,
etc.) in a way that doesn't require the implementation of these behaviors to
black-hole into the core implementations. It's necessary to implement this
because of the way that transactions must work in the SDK... simply calling
SetRDSFactory(c, wrapperFunction) is insufficient to capture changes to the
context when crossing transaction boundaries, and because context is opaque,
there's no way to reconstruct the set of filters in an implementation of
RunInTransaction. Basically what this CL does is allow RDS (or any service) to
change state in context as it likes to, and then at the point where a user
resolves RDS to an implementation (e.g. GetRDS), it wraps the RawDatastore
interface in zero or more filters. I'm planning to implement the following
filters (some of these already have CLs):
  * counter (in this CL) - mostly for testing, allows you to count how many
calls to each API feature occurred, as well as how many errors that feature
returned
  * featureBreaker (next CL) - also for testing, allows features to be
selectively broken and unbroken. moves this logic out of gae (the "Testable"
interface) and out of the memory implementation
  * mcache - for prod+testing, implements a memcache caching layer on top of RDS
(like ndb/ndb/goon)
  * txnBuffer - for prod+testing, buffers all reads/writes inside of a
transaction so that:
    * reads inside the buffered transaction reflect previous writes in the same
transaction
    * writes are delayed until the transaction is closed, preventing transaction
size errors (multiple Put's to the same entity are not collapsed by appserver).
    * currently implemented badly in tumble (since the datastore service has no
actual interface!), but can be broadly useful (in fact, I can't ever think of a
reason NOT to use it), and would fit very well as a filter

[iannucci, 2015-07-14 23:37:46]

updated doc.go to hopefully make it clearer too (also doc.go was just plain out
of date)

[iannucci, 2015-07-16 22:51:29]

iannucci@chromium.org changed reviewers:
+ estaab@chromium.org

[iannucci, 2015-07-16 22:51:29]

+estaab

any takers? :)

[estaab, 2015-07-16 22:53:26]

On 2015/07/16 22:51:29, iannucci wrote:
> +estaab
> 
> any takers? :)

I'll have a look but I'd like someone with more Go experience to review also -
Vadim?

[iannucci, 2015-07-16 22:56:50]

Patchset #15 (id:270001) has been deleted

[iannucci, 2015-07-16 22:57:11]

FWIW, I moved filters/count into another CL

[martiniss, 2015-07-16 22:58:09]

On 2015/07/16 at 22:57:11, iannucci wrote:
> FWIW, I moved filters/count into another CL

I'll take a look at it as well, although I agree with estaab.

[iannucci, 2015-07-16 23:00:47]

Actually hold off, I'm going to split this up further.

[iannucci, 2015-07-16 23:33:18]

OK, PTAL now, it's pretty pure and much shorter at this point. As a bonus I also
added tests for the copypasta code.

[Vadim Sh., 2015-07-16 23:39:51]

lgtm

thanks for splitting it in multiple CLs

https://codereview.chromium.org/1222983002/diff/400001/go/src/infra/gae/libs/...
File go/src/infra/gae/libs/gae/globalinfo.go (right):

https://codereview.chromium.org/1222983002/diff/400001/go/src/infra/gae/libs/...
go/src/infra/gae/libs/gae/globalinfo.go:64: return ret
it would have been nice to cache it somehow

[iannucci, 2015-07-16 23:46:19]

thanks for the review, sorry for starting with such a sloppy CL :)

https://chromiumcodereview.appspot.com/1222983002/diff/400001/go/src/infra/ga...
File go/src/infra/gae/libs/gae/globalinfo.go (right):

https://chromiumcodereview.appspot.com/1222983002/diff/400001/go/src/infra/ga...
go/src/infra/gae/libs/gae/globalinfo.go:64: return ret
On 2015/07/16 23:39:51, Vadim Sh. wrote:
> it would have been nice to cache it somehow

yeah, though I think that would make things more complex (and I don't expect
this method to be anything close to a hot spot in the code). We can definitely
add that later if it's an issue.

Really, the big win here would be simplifying the way that transactions work to
make them part of the gae package (interfaces) instead of part of the
implementation... I'm not sure if that's possible.

[iannucci, 2015-07-16 23:46:25]

The CQ bit was checked by iannucci@chromium.org

[commit-bot: I haz the power, 2015-07-16 23:46:54]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1222983002/400001

[commit-bot: I haz the power, 2015-07-16 23:50:26]

Committed patchset #21 (id:400001) as
https://chromium.googlesource.com/infra/infra/+/ce7d584c71c3866963c8f12baa950...