Extensions Docs Server: Apps samples page

chrome/common/extensions/docs/server2/github_file_system.py

+# Copyright (c) 2012 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import json
+import os
+
+import appengine_memcache as memcache
+import file_system
+from io import BytesIO
+from future import Future
+from zipfile import ZipFile
+
+ZIP_KEY = 'zipball'
+
+class _AsyncFetchFutureZip(object):
+  def __init__(self, fetcher, blobstore, new_version, old_version):
+    self._fetch = fetcher.FetchAsync(ZIP_KEY)
+    self._blobstore = blobstore
+    self._new_version = new_version
+    self._old_version = old_version
+
+  def Get(self):
+    self._blobstore.Delete(ZIP_KEY, self._old_version)

[not at google - send to devlin, 2012/08/07 01:35:38]

Won't blobstore also need to be namespaced (like AppengineMemcache is) for this
to not potentially result in namespace conflicts?

I guess AppengineBlobstore only uses it within its own memory and using an
AppengineMemcache instance, so if it's the same as here then it'll be safe, but
something to keep in mind when other persistent storage is being used.

[cduvall, 2012/08/08 19:27:25]

I think the version functions as a namespace for the blobstore. We also want all
the branches to have access to the same blobs, so I think this should work.

[not at google - send to devlin, 2012/08/09 07:52:36]

sgtm. I don't know so much about the versions though.

+    blob = self._fetch.Get().content
+    self._blobstore.Set(blob, ZIP_KEY, self._new_version)

[not at google - send to devlin, 2012/08/07 01:35:38]

Why are you versioning the zip files? Just curious.

[cduvall, 2012/08/08 19:27:25]

I versioned them so if there's an old zip hanging around we will be sure to get
the new one. I guess its not super necessary, so I can take it out if you want.

[not at google - send to devlin, 2012/08/09 07:52:36]

I made comment in the new version of the code.

+    return ZipFile(BytesIO(blob))
+
+class GithubFileSystem(file_system.FileSystem):
+  """FileSystem implementation which fetches resources from github.
+  """
+  def __init__(self, fetcher, memcache, blobstore):
+    self._fetcher = fetcher
+    self._memcache = memcache
+    self._blobstore = blobstore
+    self._zip_file = None
+    self._version = 0
+
+  def _GetZip(self, version):
+    blob = self._blobstore.Get(ZIP_KEY, version)
+    if blob is not None:
+      self._zip_file = Future(value=ZipFile(BytesIO(blob)))
+    else:
+      self._zip_file = Future(delegate=_AsyncFetchFutureZip(self._fetcher,
+                                                            self._blobstore,
+                                                            version,
+                                                            self._version))

[not at google - send to devlin, 2012/08/07 01:35:38]

Nice, doing this asynchronously is the way to go.

However, after calling _GetZip in Read, it either calls _ReadFile or _ListDir
which just calls _zip_file.Get() anyway, so the effort to make this asynchronous
is kinda lost.

This needs to be rearchitected slightly.

[cduvall, 2012/08/08 19:27:25]

I'm starting the fetch when the GithubFileSystem is constructed now. I'm not
sure if there's a way to take advantage of this being asynchronous after that.

[not at google - send to devlin, 2012/08/09 07:52:36]

Well... I realised there actually isn't much advantage to it anyway, since the
asynchronous wins come from when there are multiple things to get. Here there's
only one, but if we change Stat to be able to take multiple things at once...
maybe there would.

(to make this asynchronous you'd need to lift more things into Futures)

+      self._version = version
+
+  def _ReadFile(self, path):
+    zip_file = self._zip_file.Get()
+    prefix = zip_file.namelist()[0][:-1]
+    return zip_file.read(prefix + path)
+
+  def _ListDir(self, path):
+    filenames = self._zip_file.Get().namelist()
+    # Take out parent directory name (GoogleChrome-chrome-app-samples-c78a30f)
+    filenames = [f[len(filenames[0]) - 1:] for f in filenames]
+    # Remove the path of the directory we're listing from the filenames.
+    filenames = [f[len(path):] for f in filenames
+                 if f != path and f.startswith(path)]
+    # Remove all files not directly in this directory.
+    return [f for f in filenames if f[:-1].count('/') == 0]
+
+  def Read(self, paths, binary=False):
+    version = self.Stat(ZIP_KEY).version
+    if True:#self._zip_file is None or version != self._version:

[not at google - send to devlin, 2012/08/07 01:35:38]

?

[cduvall, 2012/08/08 19:27:25]

oops :)

+      self._GetZip(version)
+    result = {}
+    for path in paths:
+      if path.endswith('/'):
+        result[path] = self._ListDir(path)
+      else:
+        result[path] = self._ReadFile(path)
+    return Future(value=result)
+
+  def Stat(self, path):
+    version = self._memcache.Get(path, memcache.MEMCACHE_GITHUB_STAT)
+    if version is not None:
+      return self.StatInfo(version)
+    version = json.loads(
+        self._fetcher.Fetch('commits/HEAD').content)['commit']['tree']['sha']
+    self._memcache.Set(path, version, memcache.MEMCACHE_GITHUB_STAT)
+    return self.StatInfo(version)