sparefoot.com
robots.txt

Robots Exclusion Standard data for sparefoot.com

Resource Scan

Scan Details

Site Domain sparefoot.com
Base Domain sparefoot.com
Scan Status Ok
Last Scan2025-04-14T02:52:53+00:00
Next Scan 2025-04-21T02:52:53+00:00

Last Scan

Scanned2025-04-14T02:52:53+00:00
URL https://sparefoot.com/robots.txt
Redirect https://www.sparefoot.com/robots.txt
Redirect Domain www.sparefoot.com
Redirect Base sparefoot.com
Domain IPs 104.17.182.192, 104.17.183.192
Redirect IPs 104.18.32.64, 172.64.155.192, 2606:4700:4400::6812:2040, 2606:4700:4400::ac40:9bc0
Response IP 104.18.32.64
Found Yes
Hash 4fb2cd1732dac134812179616e3c80c33fcce92c2a779de52130bfcd820a0322
SimHash 6897c950e211

Groups

claritybot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

claritybot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

*

Rule Path Comment
Disallow /cdn-cgi/ -
Disallow /account.html -
Disallow /account/ -
Disallow /admin.html -
Disallow /admin/ -
Disallow /argus/ -
Disallow /clickheat/ -
Disallow /images/map-markers/ -
Disallow /iss/ -
Disallow /out/ -
Disallow /reviews/post_review/ -
Disallow /search/autocomplete/ -
Disallow /sitemaps/pages.html -
Disallow /widgets/ -
Disallow /reserve/ -
Disallow /*/reserve/ -
Disallow /reservation -
Disallow /sandbox/ -
Disallow /imageVerifyVisit -
Disallow /pixel -
Disallow /v1 Disallow segment proxy
Disallow /_s Disallow segment proxy

Comments

  • https://www.seoclarity.net/bot.html
  • delay is in s before a retry.
  • (250pgs * 1s) / 60 = 4.16m