Log In Sign Up

whatstheirip.tech
robots.txt

Robots Exclusion Standard data for whatstheirip.tech

Resource Scan

Scan Details

Site Domain whatstheirip.tech
Base Domain whatstheirip.tech
Scan Status Ok
Last Scan2025-10-21T02:49:31+00:00
Next Scan 2025-10-28T02:49:31+00:00

Last Scan

Scanned2025-10-21T02:49:31+00:00
URL https://whatstheirip.tech/robots.txt
Domain IPs 104.21.80.47, 172.67.174.86, 2606:4700:3035::6815:502f, 2606:4700:3035::ac43:ae56
Response IP 172.67.174.86
Found Yes
Hash c6f8f6c871fcf2b187564159ffc31c7bd5b324bd7a2093b93d80119adc2a4bae
SimHash 601cda3065a2

Groups

*

Rule Path
Allow /
Allow /blog/
Allow /blog/zh/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://whatstheirip.tech/sitemap.xml
Back to top

Comments

  • robots.txt for whatstheirip.tech
  • Updated: 2025-10-14
  • Sitemap location
  • Crawl delay for respectful crawling
  • Allow all major search engines
  • Disallow spam bots
Back to top