Chromium Code Reviews (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out

Unified Diff: appengine/monorail/static/robots.txt

Issue 1868553004: Open Source Monorail (Closed) Base URL:
Patch Set: Rebase Created 4 years, 8 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View side-by-side diff with in-line comments
Download patch
Index: appengine/monorail/static/robots.txt
diff --git a/appengine/monorail/static/robots.txt b/appengine/monorail/static/robots.txt
new file mode 100644
index 0000000000000000000000000000000000000000..3682c86179c35ae066188cb9fada52c67ba80a3c
--- /dev/null
+++ b/appengine/monorail/static/robots.txt
@@ -0,0 +1,13 @@
+User-agent: *
+# Start by disallowing everything.
+Disallow: /
+# Some specific things are okay, though.
+Allow: /$
+Allow: /hosting
+Allow: /p/*/adminIntro
+# Query strings are hard. We only allow ?id=N, no other parameters.
+Allow: /p/*/issues/detail?id=*
+Disallow: /p/*/issues/detail?id=*&*
+Disallow: /p/*/issues/detail?*&id=*
+# 10 second crawl delay for bots that honor it.
+Crawl-delay: 10

Powered by Google App Engine
This is Rietveld 408576698