Log In Sign Up

myaccount.economist.com
robots.txt

Robots Exclusion Standard data for myaccount.economist.com

Resource Scan

Scan Details

Site Domain myaccount.economist.com
Base Domain economist.com
Scan Status Ok
Last Scan2024-11-06T15:50:23+00:00
Next Scan 2024-11-20T15:50:23+00:00

Last Scan

Scanned2024-11-06T15:50:23+00:00
URL https://myaccount.economist.com/robots.txt
Domain IPs 23.215.7.19, 23.215.7.26, 2600:1413:b000:1b::17d7:713, 2600:1413:b000:1b::17d7:71a
Response IP 23.209.46.149
Found Yes
Hash 73f95b9f9e1d1d9cbff0b01eeafbc7aef2b3ffc8241bdfd9187218c98031087c
SimHash f1e065788ae7

Groups

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://myaccount.economist.com/s/sitemap.xml
sitemap https://myaccount.economist.com/economist/s/sitemap.xml
Back to top

Comments

  • GPTBot is OpenAI’s web crawler
  • Allows us to block Google's bot Bard
  • ChatGPT-User is OpenAI’s web crawler
  • Common Crawl bot
  • PiplBot is PiplBot's web crawler
  • anthropic-ai is Anthropic's web crawler
  • Claude-Web is Claude’s web crawler
  • TurnitinBot is Turnitin’s web crawler
  • PetalBot is Petal’s web crawler
  • MoodleBot is Moodle’s web crawler
  • magpie-crawler is Brandwatch.com’s web crawler
  • ia_archiver is Wayback Machine’s web crawler
  • Applebot-Extended is Apple's secondary user agent
  • PerplexityBot is the crawler for perplexity AI
  • Bytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs including those powering ChatGPT competitor Doubao.
Back to top

Warnings

  • 26 invalid lines.
  • `<!-- page generation time` is not a known field.
  • `<!doctype html public "-//w3c//dtd html 4.01 transitional//en" "http` is not a known field.
  • `</div><div class="clearingbox"></div><div class="zen"><div class="zen-pagefooter zen-pbl"><ul class="zen-pipedlist"><li class="zen-firstitem"><span class="brandquaternaryfgr">copyright © 2000-2024 salesforce.com, inc. all rights reserved.</span></li><li><a href="http` is not a known field.
  • `html .brandprimarybgr{background-color` is not a known field.
  • `html .brandprimarybrd2{border-color` is not a known field.
  • `html .brandprimarybrd{border-top-color` is not a known field.
  • `html .brandprimaryfgrbrdtop{border-top-color` is not a known field.
  • `html .brandprimaryfgr{color` is not a known field.
  • `html .brandquaternarybgr{background` is not a known field.
  • `html .brandquaternaryfgr{color` is not a known field.
  • `html .brandsecondarybgr{background-color` is not a known field.
  • `html .brandsecondarybrd{border-color` is not a known field.
  • `html .brandtertiarybgr{background-color` is not a known field.
  • `html .brandtertiarybrd{border-top-color` is not a known field.
  • `html .brandtertiaryfgr{color` is not a known field.
  • `html .brandzeronaryfgr{color` is not a known field.
  • `usercontext.initialize({"ampm"` is not a known field.
  • `var motif_key='home';</script><link rel="shortcut icon" href="https` is not a known field.
  • `}(window.uitheme = window.uitheme || {}));</script><style type="text/css">html .brandzeronarybgr{background-color` is not a known field.
Back to top