task_queues: Add more scaffolding

appengine/swarming/server/bot_management.py

Index: appengine/swarming/server/bot_management.py
diff --git a/appengine/swarming/server/bot_management.py b/appengine/swarming/server/bot_management.py
index 678af72f99e8a7055578048b4a2dfe6af36f73a8..b83821c3e7274e0b087146fd215f764603ce0333 100644
--- a/appengine/swarming/server/bot_management.py
+++ b/appengine/swarming/server/bot_management.py
@@ -9,13 +9,21 @@
           |id=bot_id|
           +---------+
                |
-        +------+-------+---------------+---------+----- ... -----+
-        |              |               |         |               |
-        v              v               v         v               v
-    +-------+    +-----------+    +--------+ +--------+     +--------+
-    |BotInfo|    |BotSettings|    |BotEvent| |BotEvent| ... |BotEvent|
-    |id=info|    |id=settings|    |id=fffff| |if=ffffe| ... |id=00000|
-    +-------+    +-----------+    +--------+ +--------+     +--------+
+        +-----++-------+---------------+---------+----- ... -----+
+        |      |        |               |         |               |
+        v      |        v               v         v               v
+    +-------+  |  +-----------+    +--------+ +--------+     +--------+
+    |BotInfo|  |  |BotSettings|    |BotEvent| |BotEvent| ... |BotEvent|
+    |id=info|  |  |id=settings|    |id=fffff| |if=ffffe| ... |id=00000|
+    +-------+  |  +-----------+    +--------+ +--------+     +--------+
+               |
+               +--------------------+
+               |                    |
+               v                    v
+    +-------------------+     +-------------------+
+    |BotTaskDimensions  | ... |BotTaskDimensions  |

[Vadim Sh., 2017/04/26 18:26:24]

do this entities created/deleted separately or always as a whole batch? Maybe
instead of keeping N entities and using ancestor keys only queries (that are NOT
cached), it will be possible to keep all dimension hashes in a single entity (as
a non-indexed repeated field) and fetch it at once (relying on ndb cache).

[M-A Ruel, 2017/04/26 21:07:07]

That's an excellent question and Michael had raised this too.

In practice I think it doesn't matter.

If I were to do one entity, this would require a transaction. The current design
doesn't require one. So this is actually faster for writes.

For reads, it keeps a cache in memcache that is coherent, so it's not a perf
problem in practice.

If it ever becomes a problem, I could switch over, this would be slightly
breaking change but not too breaking. I prefer to try to not require transaction
initially.

[Vadim Sh., 2017/04/26 21:24:55]

It will require a transaction only if we add dimension_hashes one by one (and
not, for example, rebuild them all at once from scratch). I haven't seen the
code yet.
ndb.Qurery() results are not memcached. Fetching all BotTaskDimensions entities
will always hit datastore.
ok

[M-A Ruel, 2017/04/26 21:30:30]

Actually each dimensions_hash has its own expiration time so it needs to be
updated more frequently. Anyhow, I'm keeping this in mind.
I meant I'm caching the ndb.Query result manually. It's like 5 lines.

+    |id=<dimension_hash>| ... |id=<dimension_hash>|  <task_queues.py>
+    +-------------------+     +-------------------+
 
     +--------Root--------+
     |DimensionAggregation|