snel.com
robots.txt

Robots Exclusion Standard data for snel.com

Resource Scan

Scan Details

Site Domain snel.com
Base Domain snel.com
Scan Status Ok
Last Scan2024-05-27T09:38:28+00:00
Next Scan 2024-06-10T09:38:28+00:00

Last Scan

Scanned2024-05-27T09:38:28+00:00
URL https://snel.com/robots.txt
Domain IPs 104.26.8.67, 104.26.9.67, 172.67.70.89, 2606:4700:20::681a:843, 2606:4700:20::681a:943, 2606:4700:20::ac43:4659
Response IP 104.26.8.67
Found Yes
Hash 952e15385d15f3e30e2a69e8ae4e7fff503340c7b9dfa7ed56efe1b32974cd59
SimHash 20415f010f32

Groups

*

Rule Path
Disallow /*.sql$
Disallow /*.tar$
Disallow /*.tgz$

mj12bot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bspider

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.snel.com/sitemap_index.xml
sitemap https://www.snel.com/post-sitemap.xml
sitemap https://www.snel.com/page-sitemap.xml
sitemap https://www.snel.com/kb-sitemap.xml
sitemap https://www.snel.com/kbtopic-sitemap.xml

Comments

  • Enable robots.txt rules for all crawlers
  • Sitemap
  • Do not crawl development and backup files