defineagile.com
robots.txt

Robots Exclusion Standard data for defineagile.com

Resource Scan

Scan Details

Site Domain defineagile.com
Base Domain defineagile.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-10-18T00:46:08+00:00
Next Scan 2025-10-25T00:46:08+00:00

Last Successful Scan

Scanned2025-10-03T00:45:14+00:00
URL https://defineagile.com/robots.txt
Domain IPs 104.21.73.97, 172.67.160.84, 2606:4700:3032::6815:4961, 2606:4700:3033::ac43:a054
Response IP 172.67.160.84
Found Yes
Hash 97f1d5b162e0969b58b20d9536468b52b2425595b256f168f0100edcef117362
SimHash 650809d5c7b5

Groups

googlebot

Rule Path
Allow /*

Other Records

Field Value
crawl-delay 3

*

Rule Path
Allow /*

Other Records

Field Value
sitemap https://defineagile.com/sitemap.xml