Log In Sign Up

rate-my-agent.com
robots.txt

Robots Exclusion Standard data for rate-my-agent.com

Resource Scan

Scan Details

Site Domain rate-my-agent.com
Base Domain rate-my-agent.com
Scan Status Ok
Last Scan2026-02-07T14:12:27+00:00
Next Scan 2026-03-09T14:12:27+00:00

Last Scan

Scanned2026-02-07T14:12:27+00:00
URL https://rate-my-agent.com/robots.txt
Domain IPs 104.26.0.184, 104.26.1.184, 172.67.73.207, 2606:4700:20::681a:1b8, 2606:4700:20::681a:b8, 2606:4700:20::ac43:49cf
Response IP 104.26.0.184
Found Yes
Hash 6fd232b0f167ced9bfa850c0637d643eef7c66b3ed6523424555c8920d2d37af
SimHash 4b3c5a61e154

Groups

googlebot-news
googlebot
google
bingbot
bing

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

*

Rule Path
Disallow */badge-archive$
Disallow */edit$
Disallow */show$
Disallow */reply$
Disallow */rate$
Disallow */edit_change_request$

Other Records

Field Value
crawl-delay 60
Back to top

Other Records

Field Value
sitemap https://www.rate-my-agent.com/sitemap.xml.gz
Back to top

Comments

  • AI BOT Block
  • User-agent: CCBot
  • Disallow: /
  • User-agent: ChatGPT-User
  • Disallow: /
  • User-agent: GPTBot
  • Disallow: /
  • User-agent: anthropic-ai
  • Disallow: /
  • User-agent: ClaudeBot
  • Disallow: /
  • User-agent: PetalBot
  • Disallow: /
  • User-agent: Omgilibot
  • Disallow: /
  • User-agent: Omgili
  • Disallow: /
  • User-agent: FacebookBot
  • Disallow: /
  • User-agent: Diffbot
  • Disallow: /
  • User-agent: Bytespider
  • Disallow: /
  • User-agent: ImagesiftBot
  • Disallow: /
  • User-agent: cohere-ai
  • Disallow: /
  • User-agent: Amazonbot
  • Disallow: /
Back to top