Docserver: Persist stat cache for versioned file systems

chrome/common/extensions/docs/server2/appengine_url_fetcher.py

Index: chrome/common/extensions/docs/server2/appengine_url_fetcher.py
diff --git a/chrome/common/extensions/docs/server2/appengine_url_fetcher.py b/chrome/common/extensions/docs/server2/appengine_url_fetcher.py
index 8ed0494b3d2f4832e1c8ce181c83a3fc2bb5fb2c..4dbe7436fbe98b038ba093ba9e0b18d39ad9968b 100644
--- a/chrome/common/extensions/docs/server2/appengine_url_fetcher.py
+++ b/chrome/common/extensions/docs/server2/appengine_url_fetcher.py
@@ -42,7 +42,7 @@ class AppEngineUrlFetcher(object):
     """Fetches a file synchronously.
     """
     return urlfetch.fetch(self._FromBasePath(url),
-                          deadline=20,
+                          deadline=40,

[not at google - send to devlin, 2014/10/20 21:06:57]

Do we know this is failing?

[Ken Rockot(use gerrit already), 2014/10/22 03:19:54]

Yes. I was seeing timeouts occurring at the 20 second mark. I don't know how
consistent it is, but really I also don't see any harm in letting tasks block
for a while on requests.

Alternatively we can turn this way down and let tasks fail when remote services
are unresponsive. This frees up the task queue for other stuff.

If the gitiles delays are really sporadic per request, that might make sense.
Otherwise it would probably just cause the queue to churn a lot.

[not at google - send to devlin, 2014/10/22 16:51:57]

I have a TODO somewhere to retry failed fetches, but obviously I never
implemented it. The deadline parameter should really be passed into Fetch but
20s always seemed like a safe. I think having it "low" like this might have
occasionally failed on the subversion fs as well.

Anyway - happy to increase this, though it'd be nice get some metrics on how
long these fetches take. Like, 90% complete in under 5s, 10% under 10s, etc etc.

                           headers=_MakeHeaders(username,
                                                password,
                                                access_token))
@@ -62,7 +62,7 @@ class AppEngineUrlFetcher(object):
         return self.FetchAsync(url, username, password, access_token).Get()
       return result
 
-    rpc = urlfetch.create_rpc(deadline=20)
+    rpc = urlfetch.create_rpc(deadline=40)
     urlfetch.make_fetch_call(rpc,
                              self._FromBasePath(url),
                              headers=_MakeHeaders(username,