avherald.com
robots.txt

Robots Exclusion Standard data for avherald.com

Resource Scan

Scan Details

Site Domain avherald.com
Base Domain avherald.com
Scan Status Ok
Last Scan2024-10-29T23:04:04+00:00
Next Scan 2024-11-05T23:04:04+00:00

Last Scan

Scanned2024-10-29T23:04:04+00:00
URL https://avherald.com/robots.txt
Domain IPs 62.99.163.135
Response IP 62.99.163.135
Found Yes
Hash b60cd4b8cb4b0ff8554e764203505b0722376fb0d618019a72dc1ffabb1a65a3
SimHash 2d7a9941d290

Groups

mediapartners-google

Rule Path
Disallow /cgi-bin/

googlebot

Rule Path
Disallow /cgi-bin/

google

Rule Path
Disallow /cgi-bin/

adsbot-google-mobile

Rule Path
Disallow /cgi-bin/

adsbot-google

Rule Path
Disallow /cgi-bin/

adsbot-google-mobile-apps

Rule Path
Disallow /cgi-bin/

bingbot

Rule Path
Disallow /cgi-bin/

Other Records

Field Value
crawl-delay 60

msnbot

Rule Path
Disallow /cgi-bin/

slurp

Rule Path
Disallow /cgi-bin/

*

Rule Path
Disallow /

Other Records

Field Value
sitemap https://avherald.com/h?sitemap_web=0
sitemap https://avherald.com/h?news_sitemap=0
sitemap http://avherald.com/h?sitemap_web=0
sitemap http://avherald.com/h?news_sitemap=0