dakota-fanning.org
robots.txt

Robots Exclusion Standard data for dakota-fanning.org

Resource Scan

Scan Details

Site Domain dakota-fanning.org
Base Domain dakota-fanning.org
Scan Status Ok
Last Scan2024-11-12T19:25:47+00:00
Next Scan 2024-11-19T19:25:47+00:00

Last Scan

Scanned2024-11-12T19:25:47+00:00
URL https://dakota-fanning.org/robots.txt
Domain IPs 104.21.88.247, 172.67.154.176, 2606:4700:3031::6815:58f7, 2606:4700:3037::ac43:9ab0
Response IP 172.67.154.176
Found Yes
Hash 23d9c66f5264a27a924165b22da2e1294b6a71ccef27eeb8cd5608d62256e965
SimHash 7d3ed760e6aa

Groups

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5