Extensions Docs Server: Apps samples page

chrome/common/extensions/docs/server2/zip_file_system.py

+# Copyright (c) 2012 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import json
+import os
+
+import file_system
+from io import BytesIO
+from future import Future
+from zipfile import ZipFile
+
+class ZipFileSystem(file_system.FileSystem):

[not at google - send to devlin, 2012/07/30 11:24:05]

This is super cool, though it would be nice to re-use the MemcacheFileSystem
infrastructure (which is tested etc).

Furthermore, how big are these zip files going to be? We should be careful, we
might check in a lot of data (images, sounds, etc) and memcache will start being
mean to us for trying to put a large zip file in there.

Does github expose a more granular API? Like, the ability to list files, get
their versions, read files without downloading the whole zip file? I hope so.
Then this can be like SubversionFileSystem (call it GithubFileSystem, or if it's
possible to use generic git protocols then that's even better since Chromium
will at some point migrate to git, and call it GitFileSystem).

Sorry to ask you do throw away this file... though it's only a small amount of
code :) it happens.

[cduvall, 2012/08/02 01:14:53]

So I switched to fetching individual files and directories like
SubversionFileSystem, but it is soooo slow. It can't complete a request without
timing out even on the App Engine Server. I'm pretty sure it works because I
limited the samples (like to only samples that start with 'a'), and the page
renders right, but with all the samples, it goes way too slow.

[not at google - send to devlin, 2012/08/02 13:27:48]

Ah. I wonder if that's because we're getting rate limited by github.

http://developer.github.com/v3/#rate-limiting

Ok, I guess zip file it is.

+  """FileSystem implementation which fetches resources from a zip file.
+  """
+  def __init__(self, fetcher):
+    self._fetcher = fetcher
+    self._GetZip()
+
+  def _GetZip(self):
+    self._version = self.Stat('zipball').version
+    self._zip_file = ZipFile(BytesIO(self._fetcher.Fetch('zipball')))
+
+  def _ReadFile(self, path):
+    prefix = self._zip_file.namelist()[0][:-1]
+    return self._zip_file.read(prefix + path)
+
+  def _ListDir(self, path):
+    filenames = self._zip_file.namelist()
+    # Take out parent directory name (GoogleChrome-chrome-app-samples-c78a30f)
+    filenames = [f[len(filenames[0]) - 1:] for f in filenames]
+    # Remove the path of the directory we're listing from the filenames.
+    filenames = [f[len(path):] for f in filenames
+                 if f != path and f.startswith(path)]
+    # Remove all files not directly in this directory.
+    return [f for f in filenames if f[:-1].count('/') == 0]
+
+  def Read(self, paths, binary=False):
+    if self.Stat('zipball').version != self._version:
+      self._GetZip()

[not at google - send to devlin, 2012/07/30 11:24:05]

You should try to put these expensive operations (getting the zip file from
github) inside the Future i.e. make it asynchronous.

[cduvall, 2012/08/02 01:14:53]

Done.

+    result = {}
+    for path in paths:
+      if path.endswith('/'):
+        result[path] = self._ListDir(path)
+      else:
+        result[path] = self._ReadFile(path)
+    return Future(value=result)
+
+  def Stat(self, path):
+    return self.StatInfo(json.loads(self._fetcher.Fetch('commits'))[0]['sha'])