Log In Sign Up

feedyourskull.com
robots.txt

Robots Exclusion Standard data for feedyourskull.com

Resource Scan

Scan Details

Site Domain feedyourskull.com
Base Domain feedyourskull.com
Scan Status Ok
Last Scan2024-10-21T19:35:59+00:00
Next Scan 2024-11-20T19:35:59+00:00

Last Scan

Scanned2024-10-21T19:35:59+00:00
URL https://feedyourskull.com/robots.txt
Domain IPs 69.164.195.86
Response IP 69.164.195.86
Found Yes
Hash 904f435764b60f4c2a2a602e9dc1fa14a28af7aae4899bd17a16cc077c7bef9c
SimHash c15e11506ac0

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-login.php

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

awariobot
awariorssbot
awariosmartbot

Rule Path
Disallow /

baiduspider
baiduspider-image
baiduspider-video
baiduspider-news
baiduspider-favo
baiduspider-ads
baiduspider-cpro

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

buck

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

coccocbot
coccocbot-web
coccocbot-image

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended
googlebot-image
mediapartners-google
adsbot-google

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

sogou inst spider
sogou web spider

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /
Back to top

Comments

  • Bytedance
  • Fairly certain Bytedance does not respect robots.txt, but I'm monitoring.
  • Google
  • The standard GoogleBot, used to crawl the web for their
  • search results is allowed. I don't see the value in having
  • my content consumed by the others.
Back to top