Upload file to app engine and generate public url for dmprof visualizer.

tools/deep_memory_profiler/visualizer/app.py

+# Copyright 2013 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import jinja2
+import json
+import os
+import urllib
+import webapp2
+
+from google.appengine.ext import ndb
+from google.appengine.ext import blobstore
+from google.appengine.ext.webapp import blobstore_handlers
+
+
+JINJA_ENVIRONMENT = jinja2.Environment(
+  loader=jinja2.FileSystemLoader(os.path.dirname(__file__)),
+  extensions=['jinja2.ext.autoescape'])
+
+
+class Profiler(ndb.Model):
+  """Profiler entity to store json data. Use run_id as its key.
+  Json data will be stored at blobstore, but can be referred by BlobKey which
+  can be constructed by blob_str."""
+  blob_str = ndb.StringProperty()

[sullivan, 2013/09/17 14:58:47]

Why not ndb.BlobKeyProperty?

[junjianx, 2013/09/20 06:31:09]

Done.

+
+
+class Template(ndb.Model):
+  """Template to breakdown profiler with multiple tags.
+  Use content as its key."""
+  content = ndb.PickleProperty()

[sullivan, 2013/09/17 14:58:47]

ndb.JsonProperty?

Also, isn't this redundant with json.loads(key.string_id())?

[junjianx, 2013/09/20 06:31:09]

I replaced PickleProperty with JsonProperty.
But it must be converted to string using json.dumps when constructing ndb.Key
and being passed to jinja2 template. I don't know more elegant method.

On 2013/09/17 14:58:47, sullivan wrote:

+  tags = ndb.StringProperty(repeated=True)

[sullivan, 2013/09/17 14:58:47]

Where are tags used?

[junjianx, 2013/09/20 06:31:09]

Tag will be used in next issue to help user managing templates, maybe I
shouldn't include them in this issue. Removed relative code.

On 2013/09/17 14:58:47, sullivan wrote:

+
+
+class MainPage(webapp2.RequestHandler):
+  """Show breakdown with received profiler-id and template-id. If nothing was
+  received, show blank page waiting user to upload file."""
+  def get(self):
+    page_template = JINJA_ENVIRONMENT.get_template('index.html')
+    upload_url = blobstore.create_upload_url('/upload')
+
+    # Get profiler id and template id from url query.
+    prof_id = self.request.get('prof_id')
+    tmpl_id = self.request.get('tmpl_id')
+
+    template_values = { 'upload_url': upload_url }
+
+    if not prof_id or not tmpl_id:
+      self.response.write(page_template.render(template_values))
+    else:
+      # Get entity key.
+      prof_key = ndb.Key(urlsafe=prof_id)

[sullivan, 2013/09/17 14:58:47]

It looks like it's also possible to construct a Profiler from a run_id? Not sure
if using that instead of the urlsafe form of the key would make it more
human-readable?

Also, I don't think it matters in this case, but just to be safe: It's possible
to decode the urlsafe key and easily discover the Kind and id:
https://developers.google.com/appengine/docs/python/ndb/keyclass#Key_urlsafe
This means that it would be possible for a user to modify the url to see what
profiles are in the data store. I don't see how that could harm anything, but if
they're meant to be private to the person who uploaded the profile, they're not.

[junjianx, 2013/09/20 06:31:09]

Done.

+      tmpl_key = ndb.Key(urlsafe=tmpl_id)
+
+      # Get entity object.
+      profiler = Profiler.query(ancestor=prof_key).get()
+      template = Template.query(ancestor=tmpl_key).get()
+
+      # Use blob string to construct BlobKey.
+      blob_key = blobstore.BlobKey(profiler.blob_str)
+      template_values['json'] = blobstore.BlobReader(blob_key).read()
+      template_values['template'] = template.content
+
+      self.response.write(page_template.render(template_values))
+
+
+class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
+  """Handle file uploading with BlobstoreUploadHandler. BlobstoreUploadHandler
+  can deal with files overweighing size limitation within one HTTP connection so
+  that user can upload large json file."""
+  def post(self):
+    blob_info = self.get_uploads('file')[0]
+    json_file = blob_info.open().read()
+    json_data = json.loads(json_file)
+
+    # Check the uniqueness of data run_id and store new one.
+    run_id = json_data['run_id']
+    prof_key = ndb.Key('Profiler', run_id)
+    if Profiler.query(ancestor=prof_key).count() == 0:
+      profiler = Profiler(parent=prof_key)
+      profiler.blob_str = str(blob_info.key())
+      profiler.put()

[sullivan, 2013/09/17 14:58:47]

Did you mean to have a hierarchy of Profile objects? It looks like what you
meant to do is have one Profile per run_id, all with no parents. That would look
like this:

prof_key = ndb.Key('Profiler', run_id)
if not prof_key.get()
  # Profile for this run_id does not exist
  profiler = Profiler(id=run_id, blob_key=blob_info.key())
  profiler.put()

[junjianx, 2013/09/20 06:31:09]

Done.

+
+    # Check the uniqueness of template content and store new one.
+    for tag, content in json_data['templates'].iteritems():
+      content = json.dumps(content)
+      tmpl_key = ndb.Key('Template', content)
+      if tag == json_data['default_template']:
+        default_key = tmpl_key
+      if (Template.query(ancestor=tmpl_key).count() == 0):
+        template = Template(parent=tmpl_key)
+        template.content = content
+        template.tags = [tag]
+        template.put()

[sullivan, 2013/09/17 14:58:47]

It looks like the templates are already stored in the json that goes into
blobstore. So why are they stored in the datastore as well? Any reason not to
just pull the template from the JSON in the client-side javascript, or in the
get()

Why can there be multiple templates in json_data? And is default_template
required? Looks like it fails silently if default_template is not set, and then
in get() you access the content on line 62 and it will raise an exception
accessing .content of None.

And why is there a hierarchy like there is for profile? I think you don't want
to set parents for the templates, either.

And is there a limit on the content length? I can't find the documentation on
max key length, but you probably don't want it to be more than 200 characters or
so.

[junjianx, 2013/09/20 06:31:09]

Template represents one category of breakdown. Template doesn't bind with
profiler. One template can be used to breakdown any profiler if possible.
User can create his own template (He maybe found certain breakdown graph very
interesting) and share it with others.
This is the reason we manage profiler and templates separately in database.

I think default_template should be a required attribute in json.

And we plan to add template manager in the future. User can search existing
template and add a new one.

On 2013/09/17 14:58:47, sullivan wrote:

+
+    # Jump to new graph page using default template.
+    query_params = {
+      'prof_id': prof_key.urlsafe(),
+      'tmpl_id': default_key.urlsafe()
+    }
+    self.redirect('/?' + urllib.urlencode(query_params))
+
+
+application = webapp2.WSGIApplication([
+  ('/', MainPage),
+  ('/upload', UploadHandler)
+], debug=True)