Store each file as "hash/<sha1>", list of hashes as "meta/@<rev>" and serve them at "serve_file/@<r…

gae/main.py

Index: gae/main.py
diff --git a/gae/main.py b/gae/main.py
index e9b427ccf4b140f629772d83337307455d4dc3c1..984d7530150e39e984adfa4e3e72d5786e0835cd 100644
--- a/gae/main.py
+++ b/gae/main.py
@@ -23,11 +23,32 @@ class MainPage(webapp2.RequestHandler):
     self.response.write(str(current_config))
 
 class Server(webapp2.RequestHandler):
+  def get_file_hash(self, meta_content, path):

[mnaganov (inactive), 2015/02/24 14:40:32]

We should probably cache '(tag, path) -> hash' somewhere in memory to avoid
parsing the meta file each time as well as looking up file hashes in linear
time.

[dgozman, 2015/02/25 11:37:13]

Done.

+    for line in meta_content.split('\n'):
+      index = line.find(':')
+      if index != -1 and line[(index + 1):] == path:
+        return line[:index]
+    return None
+
   def get(self, tag_type, tag, path):
     cache_key_name = '/res/%s/%s/%s' % (tag_type, tag, path)
     content = cache.get_content(cache_key_name)
     helper = config.ConfigHelper(current_config)
 
+    if not content and tag_type == 'file':

[mnaganov (inactive), 2015/02/24 14:40:32]

No, we shouldn't cache these resources under 'cache_key_name' at all. The file
reader already caches them using GCE file paths. The reason why we use
'cache_key_name' is because the contents are extracted on the fly from zip
archives. But for resources that we just read directly from GCE this will end up
creating two copies of each resource in the cache.

You should branch on 'tag_type == file' earlier, and just not access cache in
this branch -- file_reader will do caching for you.

[dgozman, 2015/02/25 11:37:13]

Done.

+      meta_file_name = helper.get_meta_path(tag)
+      meta_content = file_reader.read(helper, meta_file_name)
+      if not meta_content:
+        self.abort(404)
+      file_hash = self.get_file_hash(meta_content, path)
+      if not file_hash:
+        self.abort(404)
+      content = file_reader.read(helper, helper.get_hash_path(file_hash))
+      if not content:
+        self.abort(404)
+      content = content_processor.process(path, content)
+      cache.set_content(cache_key_name, content)
+
     if not content:
       if tag_type == 'rev':
         meta_file_name = helper.get_revision_path(tag)