Refactor the devserver to make it easier to control caching

Refactor the caching in the devserver to make it easier to configure at the top level.

Make a bunch of improvements along the way:
- Simplify cron logic by rendering all HTML files.
- Remove KnownIssuesDataSource.
- Rename MemcacheFileSystem to CachingFileSystem.
- Move a lot of Handler into ServerInstance for better dependency factoring.
- Add unit tests and test helpers for a lot of things.

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=192808

[not at google - send to devlin, 2013-04-04 03:11:49]

Hey Clark,

Apologies for how large this change ended up, but the gist of it is: change the
way that the caches are namespaced, move a lot of handler into server_instance,
and some cleanup that I found along the way.

I expect it'll need some scrutiny. I haven't yet tested on a live instance, I'll
do so after round #1.

[not at google - send to devlin, 2013-04-06 11:09:45]

Fixed a few typos and verified it works on appengine, with a few tweaks. Also
added static to the things that get fetched in the cron job, and some logging.

For what it's worth: each run of the cron job (/trunk) takes about 4 minutes.
Once I'm finally able to update to the new version of handlebar this will be
much faster (1-2 minutes I hope). The first couple of runs do time out (>10
minutes) but it stabilises after that.

cron/trunk: rendering 236 files in docs/templates/public took 233.638589859
seconds
cron/trunk: rendering 100 files in docs/static took 2.78109002113 seconds

[cduvall, 2013-04-08 07:09:24]

Awesome refactor. lgtm if it works on app engine.

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/path_canonicalizer.py (right):

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/path_canonicalizer.py:29:
'/'.join((public_template_path, 'apps')))
extra parens?

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/path_canonicalizer.py:31:
'/'.join((public_template_path, 'extensions')))
same

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/path_canonicalizer.py:41: if unix_path in
extensions_templates:
should these be checking '%s.html' % unix_path ? Or do the paths in
extension_templates not have the .html?

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/server_instance.py (right):

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/server_instance.py:47: branch_utility =
ServerInstance.branch_utility
is it necessary to assign ServerInstance.branch_utility to a new var? Could you
just do:

if ServerInstance.branch_utility is None:
  ServerInstance.branch_utility = .

[not at google - send to devlin, 2013-04-08 12:21:21]

Thanks Clark. I verified that it works on a test instance, though I can't
actually push it until I've committed and merged the change which removes known
issues.

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/path_canonicalizer.py (right):

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/path_canonicalizer.py:29:
'/'.join((public_template_path, 'apps')))
On 2013/04/08 07:09:24, cduvall wrote:
> extra parens?

Yeah, python (well, string.join) being annoying. Need to force the first
argument to be a tuple so need to double-paren it.

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/path_canonicalizer.py:31:
'/'.join((public_template_path, 'extensions')))
On 2013/04/08 07:09:24, cduvall wrote:
> same

Likewise.

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/path_canonicalizer.py:41: if unix_path in
extensions_templates:
On 2013/04/08 07:09:24, cduvall wrote:
> should these be checking '%s.html' % unix_path ? Or do the paths in
> extension_templates not have the .html?

It has an os.path.splitext which strips the html -

Though I didn't see it either, so wrote a bunch of tests, and found a bug (in
the original code I think) - so yay!

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/server_instance.py (right):

https://codereview.chromium.org/13470005/diff/13001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/server_instance.py:47: branch_utility =
ServerInstance.branch_utility
On 2013/04/08 07:09:24, cduvall wrote:
> is it necessary to assign ServerInstance.branch_utility to a new var? Could
you
> just do:
> 
> if ServerInstance.branch_utility is None:
>   ServerInstance.branch_utility = .

Done. For GithubFileSystem too.

[not at google - send to devlin, 2013-04-08 12:23:10]

Committed patchset #6 manually as r192808.