araby.ai
robots.txt

Robots Exclusion Standard data for araby.ai

Resource Scan

Scan Details

Site Domain araby.ai
Base Domain araby.ai
Scan Status Ok
Last Scan2024-05-25T10:19:20+00:00
Next Scan 2024-06-24T10:19:20+00:00

Last Scan

Scanned2024-05-25T10:19:20+00:00
URL https://araby.ai/robots.txt
Redirect https://www.araby.ai/robots.txt
Redirect Domain www.araby.ai
Redirect Base araby.ai
Domain IPs 185.179.202.145
Redirect IPs 13.35.18.14, 13.35.18.2, 13.35.18.87, 13.35.18.96, 2600:9000:20c7:4c00:16:9e11:4740:93a1, 2600:9000:20c7:5200:16:9e11:4740:93a1, 2600:9000:20c7:600:16:9e11:4740:93a1, 2600:9000:20c7:8e00:16:9e11:4740:93a1, 2600:9000:20c7:a000:16:9e11:4740:93a1, 2600:9000:20c7:ae00:16:9e11:4740:93a1, 2600:9000:20c7:ec00:16:9e11:4740:93a1, 2600:9000:20c7:fc00:16:9e11:4740:93a1
Response IP 13.35.18.87
Found Yes
Hash 4352b622dd857fc842b646e942c10100f642c63e6269b9ce75c20c09c0797a9d
SimHash 4850cb024772

Groups

*

Rule Path
Disallow /accounts

*

Rule Path
Allow /

Comments

  • Block all crawlers for /accounts
  • Allow all crawlers