Monitoring proxy for time series data

appengine/chrome_infra_mon_proxy/common.py

Index: appengine/chrome_infra_mon_proxy/common.py
diff --git a/appengine/chrome_infra_mon_proxy/common.py b/appengine/chrome_infra_mon_proxy/common.py
new file mode 100644
index 0000000000000000000000000000000000000000..5bc79ce6b89d55849cd2f9be6a1ccf6027e4d03e
--- /dev/null
+++ b/appengine/chrome_infra_mon_proxy/common.py
@@ -0,0 +1,55 @@
+# Copyright 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import hashlib
+import logging
+import os
+
+from google.appengine.ext import ndb
+
+
+CREDENTIALS_KEY = 'credentials_key'

[agable, 2015/04/14 21:46:25]

Either this should be 'data_key' or the object should be 'MonAcqCredentials'

[Sergey Berezin (google), 2015/04/16 04:39:08]

Renamed to config_data_key.

+
+
+def is_development_server():
+  return os.environ.get('SERVER_SOFTWARE', '').startswith('Development')
+
+
+class MonAcqData(ndb.Model):

[agable, 2015/04/14 21:46:24]

I would have a wholly separate datastore model just for the json p8 credentials.
That way we can just create a new datastore entry when the credentials are
updated, and keep the old ones around so we can see the history of updates if we
want to. Updating the credentials shouldn't modify just a portion of this global
config object.

That said, this object seems to mostly just be a credentials object, not a
general data object, so maybe just the name and update mechanism could be
changed.

[Sergey Berezin (google), 2015/04/16 04:39:08]

I'd postpone implementing a history of updates to a later CL. This one is
already big as it is :-)
Also, I need all the data from this record in every single request, and making
two calls to the datastore is slower than one, even if it's memcached.

But if I miss some other design consideration, please let me know.
Credentials is only one field; it also holds the endpoint URL, the scopes, and
the headers. I consider it a complete config set.

+  """Store the sensitive endpoint data."""
+  credentials = ndb.JsonProperty()
+  url = ndb.StringProperty()
+  scopes = ndb.StringProperty(repeated=True)
+  headers = ndb.JsonProperty(default={})
+
+
+def get_data():
+  data_entity = MonAcqData.get_by_id(CREDENTIALS_KEY)
+  logging.info('get_data(): entity = %r', data_entity)

[agable, 2015/04/14 21:46:24]

Seems inconsistent to log when we get_data, but not when the admin handler
updates the data.

[Sergey Berezin (google), 2015/04/16 04:39:08]

Added logging to admin_handler.

Also, moved this function to vm_module.py.

+  if not data_entity:
+    return None
+  return data_entity.to_dict()
+
+
+def get_credentials(credentials_dict, scopes):
+  """Obtain Aquisition API credentials as Credentials object."""
+  from oauth2client.client import SignedJwtAssertionCredentials
+
+  # Ideally, we should have loaded credentials with GoogleCredentials.
+  # However, it insists to load only from a file. So, here's a hack.

[agable, 2015/04/14 21:46:25]

Why can't we read the json credentials from the datastore into a CStringIO
StringIO object, which behaves like a file, and let GoogleCredentials read from
that? Does it detect that and complain? Python appengine does have the StringIO
library available.

[Sergey Berezin (google), 2015/04/16 04:39:08]

GoogleCredentials expects a file path as string... :-)

BTW, I moved this function to vm_module.py.

+  return SignedJwtAssertionCredentials(
+      service_account_name=credentials_dict['client_email'],
+      private_key=credentials_dict['private_key'],
+      scope=scopes,
+      # Extra **kwargs, just in case.
+      service_account_id=credentials_dict['client_id'],
+      private_key_id=credentials_dict['private_key_id'],
+  )
+
+
+def payload_stats(data):
+  md5 = hashlib.md5()
+  md5.update(data)
+  md5hex = md5.hexdigest()
+  return 'type=%s, %d bytes, md5=%s' % (type(data), len(data), md5hex)