insightful.systems
robots.txt

Robots Exclusion Standard data for insightful.systems

Resource Scan

Scan Details

Site Domain insightful.systems
Base Domain insightful.systems
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-08-19T22:57:20+00:00
Next Scan 2025-11-17T22:57:20+00:00

Last Successful Scan

Scanned2025-03-30T17:21:50+00:00
URL https://insightful.systems/robots.txt
Domain IPs 104.21.42.43, 172.67.156.26, 2606:4700:3034::6815:2a2b, 2606:4700:3034::ac43:9c1a
Response IP 172.67.156.26
Found Yes
Hash 210e3205e0783d27b03b59bc65a185ff75882d377e02c6c1e24a769afde36128
SimHash 901683297bb1

Groups

*

Rule Path
Disallow /tags/
Disallow /tags/*
Disallow /proxy/*

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

Comments

  • Don't allow any bot to crawl tags.
  • Don't allow bots to crawl through the proxy