Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(1814)

Unified Diff: appengine/swarming/handlers_backend.py

Issue 2258103002: Further limit TaskTags aggregation cron job to avoid OOM (Closed) Base URL: https://chromium.googlesource.com/external/github.com/luci/luci-py@master
Patch Set: Properly handle None Created 4 years, 4 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View side-by-side diff with in-line comments
Download patch
« no previous file with comments | « no previous file | no next file » | no next file with comments »
Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
Index: appengine/swarming/handlers_backend.py
diff --git a/appengine/swarming/handlers_backend.py b/appengine/swarming/handlers_backend.py
index 82659a01fc8d2328bac0f8f22dc380da32de28e0..c232647dbce617fe267b2b6df9c2ca04a71a28ff 100644
--- a/appengine/swarming/handlers_backend.py
+++ b/appengine/swarming/handlers_backend.py
@@ -144,14 +144,19 @@ class CronTasksTagsAggregationHandler(webapp2.RequestHandler):
for t in tasks:
for i in t.tags:
k, v = i.split(':', 1)
- if k != 'cron_invocation_id':
- seen.setdefault(k, set()).add(v)
+ s = seen.setdefault(k, set())
+ if s is not None:
+ s.add(v)
+ # 128 is arbitrary large number to avoid OOM
+ if len(s) >= 128:
+ logging.info('Stripping tag %s because there are too many', k)
+ seen[k] = None
if not cursor or len(tasks) == 0:
break
tags = [
task_result.TagValues(tag=k, values=sorted(values))
- for k, values in sorted(seen.iteritems())
+ for k, values in sorted(seen.iteritems()) if values is not None
]
logging.info('From %d tasks, saw tags %s', count, tags)
« no previous file with comments | « no previous file | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698