denverite.com
robots.txt

Robots Exclusion Standard data for denverite.com

Resource Scan

Scan Details

Site Domain denverite.com
Base Domain denverite.com
Scan Status Ok
Last Scan2024-10-16T10:59:06+00:00
Next Scan 2024-11-15T10:59:06+00:00

Last Scan

Scanned2024-10-16T10:59:06+00:00
URL https://denverite.com/robots.txt
Domain IPs 104.22.20.66, 104.22.21.66, 172.67.24.164, 2606:4700:10::6816:1442, 2606:4700:10::6816:1542, 2606:4700:10::ac43:18a4
Response IP 104.22.21.66
Found Yes
Hash fe6253ca0156225fc88632d30593644f968eb77d92abbfacaa9f82c7c70f7120
SimHash 2b80c8008bb3

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /*?refresh=
Disallow /newsletters/

megaindex.ru

Rule Path
Disallow /

Other Records

Field Value
sitemap https://denverite.com/sitemap.xml
sitemap https://denverite.com/news-sitemap.xml

Comments

  • Kill bad bots!
  • https://megaindex.com/crawler