Log In Sign Up

beabetterself.com
robots.txt

Robots Exclusion Standard data for beabetterself.com

Resource Scan

Scan Details

Site Domain beabetterself.com
Base Domain beabetterself.com
Scan Status Ok
Last Scan2026-01-08T18:42:43+00:00
Next Scan 2026-02-07T18:42:43+00:00

Last Scan

Scanned2026-01-08T18:42:43+00:00
URL https://beabetterself.com/robots.txt
Domain IPs 23.226.60.102
Response IP 23.226.60.102
Found Yes
Hash 1aeae96e368545a300633129b7ecca049d6d8caaa39883e72b64677c924bed6e
SimHash 5132d5c8f4b2

Groups

baiduspider

Rule Path
Allow /

sogou spider

Rule Path
Allow /

sosospider

Rule Path
Allow /

googlebot

Rule Path
Allow /

mediapartners-google

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

zmeu phpmyadmin

Rule Path
Disallow /

winhttp

Rule Path
Disallow /

httpclient

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

yyspider

Rule Path
Disallow /

jaunty

Rule Path
Disallow /

obot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

indy library

Rule Path
Disallow /

flightdeckreports bot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ezooms.bot

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

crawldaddy

Rule Path
Disallow /

asktbfxtv

Rule Path
Disallow /

alexa toolbar

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

coolpadwebkit

Rule Path
Disallow /

java

Rule Path
Disallow /

feedly

Rule Path
Disallow /

universalfeedparser

Rule Path
Disallow /

apachebench

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

lightdeckreports bot

Rule Path
Disallow /

digext

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

headlesschrome

Rule Path
Disallow /

feeddemon

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

opensiteexplorer

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

hubspot

Rule Path
Disallow /

opensiteexplorer

Rule Path
Disallow /

python

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /
Back to top

Warnings

  • 2 invalid lines.
Back to top