indietraveller.co
robots.txt

Robots Exclusion Standard data for indietraveller.co

Resource Scan

Scan Details

Site Domain indietraveller.co
Base Domain indietraveller.co
Scan Status Ok
Last Scan2024-11-12T21:10:20+00:00
Next Scan 2024-11-19T21:10:20+00:00

Last Scan

Scanned2024-11-12T21:10:20+00:00
URL https://indietraveller.co/robots.txt
Redirect https://www.indietraveller.co/robots.txt
Redirect Domain www.indietraveller.co
Redirect Base indietraveller.co
Domain IPs 162.159.136.54, 162.159.137.54, 2606:4700:7::a29f:8836, 2606:4700:7::a29f:8936
Redirect IPs 162.159.136.54, 162.159.137.54, 2606:4700:7::a29f:8836, 2606:4700:7::a29f:8936
Response IP 162.159.136.54
Found Yes
Hash e240e44bbeba3c7a0a10d43f6bc89a6e295c411149eab2bf72cb2a5efa2dbf13
SimHash 3054d9408513

Groups

*

Rule Path
Disallow /files/

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.indietraveller.co/sitemap_index.xml