mybetterhalf.com
robots.txt

Robots Exclusion Standard data for mybetterhalf.com

Resource Scan

Scan Details

Site Domain mybetterhalf.com
Base Domain mybetterhalf.com
Scan Status Ok
Last Scan2025-09-04T21:33:23+00:00
Next Scan 2025-10-04T21:33:23+00:00

Last Scan

Scanned2025-09-04T21:33:23+00:00
URL https://mybetterhalf.com/robots.txt
Redirect https://www.mybetterhalf.com/robots.txt
Redirect Domain www.mybetterhalf.com
Redirect Base mybetterhalf.com
Domain IPs 104.21.25.131, 172.67.134.65, 2606:4700:3036::6815:1983, 2606:4700:3037::ac43:8641
Redirect IPs 104.21.25.131, 172.67.134.65, 2606:4700:3036::6815:1983, 2606:4700:3037::ac43:8641
Response IP 172.67.134.65
Found Yes
Hash 0d755ddc5ff0c3d00cf7173c5dedce4b6a77779c0b898178391e9deb0bc4af9e
SimHash 405cd2e25791

Groups

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

slurp

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

baiduspider

Rule Path
Disallow

yandex

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

*

Rule Path
Disallow /site/index
Disallow /site/morevideos
Disallow /site/password
Disallow /site/subscribe
Disallow /site/sitelogin
Disallow /site/share-profile

Comments

  • Allow major search engines
  • Block known spammy/hammer bots