Log In Sign Up

strongtie.com
robots.txt

Robots Exclusion Standard data for strongtie.com

Resource Scan

Scan Details

Site Domain strongtie.com
Base Domain strongtie.com
Scan Status Ok
Last Scan2024-06-22T02:19:57+00:00
Next Scan 2024-07-22T02:19:57+00:00

Last Scan

Scanned2024-06-22T02:19:57+00:00
URL https://strongtie.com/robots.txt
Redirect https://www.strongtie.com/robots.txt
Redirect Domain www.strongtie.com
Redirect Base strongtie.com
Domain IPs 40.121.213.51
Redirect IPs 104.81.138.113, 104.81.138.8
Response IP 23.59.168.137
Found Yes
Hash 30ed74b80452014098fc4f9962424a9a0560f1b649c3635b49e4b8086f549d0b
SimHash 7844d71dedf0

Groups

*

Rule Path
Allow /

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • For all robots
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Twitter
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top