dsrinternational.com
robots.txt

Robots Exclusion Standard data for dsrinternational.com

Resource Scan

Scan Details

Site Domain dsrinternational.com
Base Domain dsrinternational.com
Scan Status Ok
Last Scan2025-11-27T16:32:13+00:00
Next Scan 2025-12-04T16:32:13+00:00

Last Scan

Scanned2025-11-27T16:32:13+00:00
URL https://dsrinternational.com/robots.txt
Domain IPs 104.21.37.163, 172.67.210.147, 2606:4700:3034::ac43:d293, 2606:4700:3037::6815:25a3
Response IP 172.67.210.147
Found Yes
Hash 56e96f526fbb86e23bca663d45e83058eefd12ee2a0e6ddaac3af3b9434cbab8
SimHash 28f04ac0a603

Groups

*

Rule Path
Disallow /admin/
Disallow /login/
Disallow /cgi-bin/
Allow /

Other Records

Field Value
sitemap https://www.dsrinternational.com/sitemap.xml

Comments

  • Allow all crawlers
  • Disallow admin or sensitive pages (example)
  • Allow everything else
  • Sitemap location (important for SEO)