Log In Sign Up

surfindustries.com
robots.txt

Robots Exclusion Standard data for surfindustries.com

Resource Scan

Scan Details

Site Domain surfindustries.com
Base Domain surfindustries.com
Scan Status Ok
Last Scan2024-09-20T19:20:52+00:00
Next Scan 2024-10-20T19:20:52+00:00

Last Scan

Scanned2024-09-20T19:20:52+00:00
URL https://www.surfindustries.com/robots.txt
Domain IPs 104.20.6.22, 104.20.7.22, 172.67.0.43, 2606:4700:10::6814:616, 2606:4700:10::6814:716, 2606:4700:10::ac43:2b
Response IP 172.67.0.43
Found Yes
Hash b331550e7fceb1b01771675388d1efccacd931cb91e695e2e7d7a068158857e5
SimHash 2b23c91757b5

Groups

*

Rule Path
Disallow /

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
yahoo-mmcrawler

Rule Path
Disallow /admin/
Disallow /brian/
Disallow /dealer-admin/
Disallow /distributor-portal/
Disallow /doug/
Disallow /newsletter/
Disallow /shop/au/*
Disallow /shop/nz/*
Disallow /shop/us/*
Disallow /TM-portal
Allow /
Back to top

Other Records

Field Value
sitemap https://www.surfindustries.com/sitemap.xml
Back to top

Comments

  • robots.txt file for https://www.surfindustries.com/
  • added by Global Surf Industries 26-06-17
  • disallow all
  • TK commented out crawl delay as per request form MS 22/5/18
  • Crawl-delay: 600
  • but allow only important bots
  • end of file
Back to top