atanews.net
robots.txt

Robots Exclusion Standard data for atanews.net

Resource Scan

Scan Details

Site Domain atanews.net
Base Domain atanews.net
Scan Status Ok
Last Scan2024-06-02T18:22:34+00:00
Next Scan 2024-07-02T18:22:34+00:00

Last Scan

Scanned2024-06-02T18:22:34+00:00
URL https://atanews.net/robots.txt
Domain IPs 104.21.13.86, 172.67.132.165, 2606:4700:3033::6815:d56, 2606:4700:3033::ac43:84a5
Response IP 104.21.13.86
Found Yes
Hash 51711b8653da3dbd8f475043757a3905b89a299457c8f4549d3230391654a427
SimHash b8051b606272

Groups

*

Rule Path
Disallow /cgi-bin

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

Comments

  • Stop crawling after 5 minutes