donorschoose.org
robots.txt

Groups

*

Other Records

Comments

• This is the production robots.txt file that allows all robots to crawl (with guidance to avoid getting trapped).
• This will only work for well-behaved robots, so it may be necessary to escalate to
• 403 Forbidden responses in the redirects.conf file.
• Block some UX content that shouldn't be in search engines.
• Robots should never create shopping carts:
• Robots should never crawl our sorry pages during BRS outages:
• Robots should never create search-like pages that aren't search.html:
• KID-GLOVE RULES FOR CHALLENGE PAGES
• Allow only id=* /donors/viewChallenge.html pages.
• Do not allow multiple params on /donors/viewChallenge.html pages
• Disallow params with an initial letter other than i.
• We should only allow id=*, so treat other i* carefully.
• DO NOT Disallow: /donors/viewChallenge.html?id=
• Disallow first 2 letters i_ where _ other than d.
• KID-GLOVE RULES FOR SEARCH PAGES
• Do not allow any views on /donors/search.html pages, except for landing pages
• Sitemaps