shodan.io
robots.txt

Robots Exclusion Standard data for shodan.io

Resource Scan

Scan Details

Site Domain shodan.io
Base Domain shodan.io
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-10T10:40:13+00:00
Next Scan 2024-10-08T10:40:13+00:00

Last Successful Scan

Scanned2023-05-25T02:59:33+00:00
URL https://shodan.io/robots.txt
Redirect https://www.shodan.io/robots.txt
Redirect Domain www.shodan.io
Redirect Base shodan.io
Domain IPs 104.18.12.238, 104.18.13.238, 2606:4700::6812:cee, 2606:4700::6812:dee
Redirect IPs 104.18.12.238, 104.18.13.238, 2606:4700::6812:cee, 2606:4700::6812:dee
Response IP 104.18.13.238
Found Yes
Hash d5f60044bba161a12081f3836b5142e08407028a88f803e2dd61eaf75c5eb826
SimHash 6150dc544712

Groups

*

Rule Path
Disallow /domain/

Other Records

Field Value
crawl-delay 10