appengine/monorail/static/robots.txt - Issue 1868553004: Open Source Monorail

Keyboard Shortcuts

	File
u :	up to issue
j / k :	jump to file after / before current file
J / K :	jump to next file with a comment after / before current file
	Side-by-side diff
i :	toggle intra-line diffs
e :	expand all comments
c :	collapse all comments
s :	toggle showing all comments
n / p :	next / previous diff chunk or comment
N / P :	next / previous comment
<Up> / <Down> :	next / previous line

	Issue
u :	up to list of issues
j / k :	jump to patch after / before current patch
o / <Enter> :	open current patch in side-by-side view
i :	open current patch in unified diff view

	Issue List
j / k :	jump to issue after / before current issue
o / <Enter> :	open current issue

Unified Diff: appengine/monorail/static/robots.txt

Issue 1868553004: Open Source Monorail (Closed) Base URL: https://chromium.googlesource.com/infra/infra.git@master

Patch Set: Rebase Created 4 years, 8 months ago

Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.

Jump to:

Index: appengine/monorail/static/robots.txt

diff --git a/appengine/monorail/static/robots.txt b/appengine/monorail/static/robots.txt

new file mode 100644

index 0000000000000000000000000000000000000000..3682c86179c35ae066188cb9fada52c67ba80a3c

--- /dev/null

+++ b/appengine/monorail/static/robots.txt

@@ -0,0 +1,13 @@

+User-agent: *

+# Start by disallowing everything.

+Disallow: /

+# Some specific things are okay, though.

+Allow: /$

+Allow: /hosting

+Allow: /p/*/adminIntro

+# Query strings are hard. We only allow ?id=N, no other parameters.

+Allow: /p/*/issues/detail?id=*

+Disallow: /p/*/issues/detail?id=*&*

+Disallow: /p/*/issues/detail?*&id=*

+# 10 second crawl delay for bots that honor it.

+Crawl-delay: 10