task_queues: Add more scaffolding

appengine/swarming/server/bot_management.py

 # Copyright 2014 The LUCI Authors. All rights reserved.
 # Use of this source code is governed under the Apache License, Version 2.0
 # that can be found in the LICENSE file.
 
 """Swarming bot management, e.g. list of known bots and their state.
 
           +---------+
           |BotRoot  |
           |id=bot_id|
           +---------+
                |
-        +------+-------+---------------+---------+----- ... -----+
-        |              |               |         |               |
-        v              v               v         v               v
-    +-------+    +-----------+    +--------+ +--------+     +--------+
-    |BotInfo|    |BotSettings|    |BotEvent| |BotEvent| ... |BotEvent|
-    |id=info|    |id=settings|    |id=fffff| |if=ffffe| ... |id=00000|
-    +-------+    +-----------+    +--------+ +--------+     +--------+
+        +-----++-------+---------------+---------+----- ... -----+
+        |      |        |               |         |               |
+        v      |        v               v         v               v
+    +-------+  |  +-----------+    +--------+ +--------+     +--------+
+    |BotInfo|  |  |BotSettings|    |BotEvent| |BotEvent| ... |BotEvent|
+    |id=info|  |  |id=settings|    |id=fffff| |if=ffffe| ... |id=00000|
+    +-------+  |  +-----------+    +--------+ +--------+     +--------+
+               |
+               +--------------------+
+               |                    |
+               v                    v
+    +-------------------+     +-------------------+
+    |BotTaskDimensions  | ... |BotTaskDimensions  |

[Vadim Sh., 2017/04/26 18:26:24]

do this entities created/deleted separately or always as a whole batch? Maybe
instead of keeping N entities and using ancestor keys only queries (that are NOT
cached), it will be possible to keep all dimension hashes in a single entity (as
a non-indexed repeated field) and fetch it at once (relying on ndb cache).

[M-A Ruel, 2017/04/26 21:07:07]

That's an excellent question and Michael had raised this too.

In practice I think it doesn't matter.

If I were to do one entity, this would require a transaction. The current design
doesn't require one. So this is actually faster for writes.

For reads, it keeps a cache in memcache that is coherent, so it's not a perf
problem in practice.

If it ever becomes a problem, I could switch over, this would be slightly
breaking change but not too breaking. I prefer to try to not require transaction
initially.

[Vadim Sh., 2017/04/26 21:24:55]

It will require a transaction only if we add dimension_hashes one by one (and
not, for example, rebuild them all at once from scratch). I haven't seen the
code yet.
ndb.Qurery() results are not memcached. Fetching all BotTaskDimensions entities
will always hit datastore.
ok

[M-A Ruel, 2017/04/26 21:30:30]

Actually each dimensions_hash has its own expiration time so it needs to be
updated more frequently. Anyhow, I'm keeping this in mind.
I meant I'm caching the ndb.Query result manually. It's like 5 lines.

+    |id=<dimension_hash>| ... |id=<dimension_hash>|  <task_queues.py>
+    +-------------------+     +-------------------+
 
     +--------Root--------+
     |DimensionAggregation|
     |id='current'        |
     +--------------------+
 
 - BotEvent is a monotonically inserted entity that is added for each event
   happening for the bot.
 - BotInfo is a 'dump-only' entity used for UI, it permits quickly show the
   state of every bots in an single query. It is basically a cache of the last
@@ skipping 427 matching lines @@
   Returns:
     Tuple (True to restart, text message explaining the reason).
   """
   # Periodically reboot bots to workaround OS level leaks (especially on Win).
   running_time = state.get('running_time', 0)
   assert isinstance(running_time, (int, float))
   period = get_bot_reboot_period(bot_id, state)
   if period and running_time > period:
     return True, 'Periodic reboot: running longer than %ds' % period
   return False, ''