Log In Sign Up

myip-address.io
robots.txt

Robots Exclusion Standard data for myip-address.io

Resource Scan

Scan Details

Site Domain myip-address.io
Base Domain myip-address.io
Scan Status Ok
Last Scan2025-09-29T00:59:17+00:00
Next Scan 2025-10-29T00:59:17+00:00

Last Scan

Scanned2025-09-29T00:59:17+00:00
URL https://myip-address.io/robots.txt
Domain IPs 104.21.47.209, 172.67.172.170, 2606:4700:3032::6815:2fd1, 2606:4700:3034::ac43:acaa
Response IP 104.21.47.209
Found Yes
Hash 24329f7c57bbea7e3d407cd4d2b1ffe7984d49de0dc7cf5d4a503e26371d6125
SimHash 503c8156cbfb

Groups

*

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

*

Rule Path
Disallow /admin/
Disallow /login/
Disallow /user/

*

Rule Path
Disallow /private/
Disallow /confidential/
Disallow /temporary/

*

Rule Path
Disallow /*?*
Back to top

Comments

  • Allow all web crawlers access to all content
  • Block Ahrefs
  • Block Majestic
  • Block Moz
  • Block Semrush
  • Additional common settings for other bots
  • Block all crawlers from accessing the admin area
  • Block all crawlers from accessing specific files (example files, adjust as needed)
  • Block all crawlers from accessing query parameters
Back to top