Count instances managed by each instance template

appengine/gce-backend/metrics.py

Index: appengine/gce-backend/metrics.py
diff --git a/appengine/gce-backend/metrics.py b/appengine/gce-backend/metrics.py
index 0b4bf227be14f37c99f68d5bc6e0ccd7259fbd05..4be7545ab366d9de2d123fc6c4e234583d58fe31 100644
--- a/appengine/gce-backend/metrics.py
+++ b/appengine/gce-backend/metrics.py
@@ -9,6 +9,7 @@ import logging
 import gae_event_mon
 import gae_ts_mon
 
+import instance_group_managers
 import instances
 
 
@@ -28,6 +29,10 @@ GLOBAL_METRICS = {
         'machine_provider/gce_backend/instances',
         description='Current count of the number of instances.',
     ),
+    'instances/v2': gae_ts_mon.GaugeMetric(
+        'machine_provider/gce_backend/instances/v2',

[Sergey Berezin (google), 2016/11/23 02:28:25]

I'd name it in a more self-descriptive way, e.g. .../instances_by_template.
Or, even better, just add a new field to the existing metric and count
orphaned/non-orphaned x templates.

[smut, 2016/11/23 22:50:57]

I tried modifying the existing metric and the metric stopped showing up at all.
I thought you once said that field schema shouldn't be changed?

At any rate, the orphaned/unorphaned distinction is somewhat meaningless. An
implementation detail causes tons of orphaned instances to appear in the minutes
following instance deletion. Originally I was counting orphans to find errors,
but now it's standard operating procedure to detect orphans.

[Sergey Berezin (google), 2016/11/23 23:42:44]

No, the field schema should be consistent globally - at least that was a
requirement previously, and apparently now missing fields don't (always) cause
the metric to fail. I'm pretty sure it's OK to change the fields if you redeploy
all the instances that send the metric. Some data may be lost for the transition
period though.
Do you care about the current history of this metric? If not, I'd say let's
update the metric with the new field, drop the 'orphaned', and once deployed,
we'll delete the metric on backend and it will be recreated in its new form. The
reason is, it's a very good name that otherwise would be waisted and be
confusing to everyone. Best to reuse it if we can. WDYT?

[smut, 2016/11/23 23:51:09]

No, I don't care about the history. I would prefer this be the only "instances"
metric. Uploaded new patch, and deployed on staging and prod. Should be safe to
delete the old metric's history now and let the new one populate.

[Sergey Berezin (google), 2016/11/24 03:51:47]

Great, I deleted the metric - once the update propagates, we'll hopefully see
just one metric field. Thanks!

+        description='Current count of the number of instances.',
+    ),
 }
 
 
@@ -53,6 +58,15 @@ def compute_global_metrics():
       },
       target_fields=GLOBAL_TARGET_FIELDS,
   )
+  for name, count in instance_group_managers.count_instances().iteritems():
+    logging.info('%s: %s', name, count)
+    GLOBAL_METRICS['instances/v2'].set(
+        count,
+        fields={
+            'instance_template': name,
+        },
+        target_fields=GLOBAL_TARGET_FIELDS,
+    )
 
 
 def initialize():