Log In Sign Up

eiu.com
robots.txt

Robots Exclusion Standard data for eiu.com

Resource Scan

Scan Details

Site Domain eiu.com
Base Domain eiu.com
Scan Status Ok
Last Scan2024-10-17T07:16:29+00:00
Next Scan 2024-11-16T07:16:29+00:00

Last Scan

Scanned2024-10-17T07:16:29+00:00
URL https://eiu.com/robots.txt
Redirect http://www.eiu.com/robots.txt
Redirect Domain www.eiu.com
Redirect Base eiu.com
Domain IPs 104.18.40.194, 172.64.147.62
Redirect IPs 104.18.40.194, 172.64.147.62
Response IP 104.18.40.194
Found Yes
Hash 1ada9ba6cad18d777cff497708ad6451619480a02497b0c235afe1e8ac937365
SimHash 51b6b970c4f1

Groups

*

Rule Path
Disallow /contents/
Disallow /images/
Disallow /graphics/
Disallow /upload/
Disallow /asset_images/
Disallow /search.asp
Disallow /search/
Disallow /search.asp
Disallow /search/
Disallow *.aspx_*
Disallow *.aspx/_*
Disallow /*/article/*2_*
Allow /AllCountries.aspx

Other Records

Field Value
crawl-delay 20

googlebot

Rule Path
Disallow /search.asp
Disallow /search/
Disallow *.aspx_*
Disallow *.aspx/_*
Disallow /*/article/*2_*
Allow /AllCountries.aspx

bingbot

Rule Path
Disallow /search.asp
Disallow /search/
Disallow *.aspx_*
Disallow *.aspx/_*
Disallow /*/article/*2_*
Allow /AllCountries.aspx

Other Records

Field Value
crawl-delay 20

slurp

Rule Path
Disallow /search.asp
Disallow /search/
Disallow *.aspx_*
Disallow *.aspx/_*
Disallow /*/article/*2_*
Allow /AllCountries.aspx

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.eiu.com/sitemap.xml
Back to top

Comments

  • GPTBot is OpenAI’s web crawler
  • Allows us to block Google's bot Bard
  • ChatGPT-User is OpenAI’s web crawler
  • Common Crawl bot
  • PiplBot is PiplBot's web crawler
  • anthropic-ai is Anthropic's web crawler
  • Claude-Web is Claude’s web crawler
  • TurnitinBot is Turnitin’s web crawler
  • PetalBot is Petal’s web crawler
  • MoodleBot is Moodle’s web crawler
  • magpie-crawler is Brandwatch.com’s web crawler
  • ia_archiver is Wayback Machine’s web crawler
  • Applebot-Extended is Apple's secondary user agent
  • PerplexityBot is the crawler for perplexity AI
Back to top

Warnings

  • `noindex` is not a known field.
Back to top