Log In Sign Up

bensbites.com
robots.txt

Robots Exclusion Standard data for bensbites.com

Resource Scan

Scan Details

Site Domain bensbites.com
Base Domain bensbites.com
Scan Status Ok
Last Scan2024-10-28T23:16:06+00:00
Next Scan 2024-11-04T23:16:06+00:00

Last Scan

Scanned2024-10-28T23:16:06+00:00
URL https://bensbites.com/robots.txt
Domain IPs 104.18.20.115, 104.18.21.115, 2606:4700::6812:1473, 2606:4700::6812:1573
Response IP 104.18.21.115
Found Yes
Hash 99b898ec7dbe7a32549ab6205015dce3e40c255cd89cfd0faed221f00db79985
SimHash 495efb70e492

Groups

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googleother

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

seekr

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

youbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

bw/1.1

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /assets/

baiduspider

Rule Path
Disallow /assets/

petalbot

Rule Path
Disallow /assets/

yandexbot

Rule Path
Disallow /assets/

ahrefsbot

Rule Path
Disallow /assets/

ahrefssiteaudit

Rule Path
Disallow /assets/

semrushbot

Rule Path
Disallow /assets/
Back to top

Other Records

Field Value
sitemap https://bensbites.com/sitemap-index.xml
Back to top

Comments

  • Block Open AI
  • Block Anthropic (Claude)
  • Block Google (Gemini)
  • Block CommonCrawl
  • Block Apple
  • Block Amazon (Alexa)
  • Block Meta
  • Block ByteDance (TikTok)
  • Block Diffbot
  • Block Webz.io
  • Block ImagesiftBot
  • Block Meltwater
  • Block Seekr
  • Block Perplexity
  • Block Cohere
  • Block You.com
  • Block ZoomInfo
  • Block BuiltWith
  • Block IONOS
  • Block DuckDuckGo from Crawling Assets
  • Block Baidu from Crawling Assets
  • Block Petal from Crawling Assets
  • Block Yandex from Crawling Assets
  • Block Ahrefs from Crawling Assets
  • Block Semrush from Crawling Assets
Back to top