turningleftforless.com
robots.txt

Robots Exclusion Standard data for turningleftforless.com

Resource Scan

Scan Details

Site Domain turningleftforless.com
Base Domain turningleftforless.com
Scan Status Ok
Last Scan2025-08-10T23:07:46+00:00
Next Scan 2025-08-17T23:07:46+00:00

Last Scan

Scanned2025-08-10T23:07:46+00:00
URL https://turningleftforless.com/robots.txt
Redirect https://www.turningleftforless.com/robots.txt
Redirect Domain www.turningleftforless.com
Redirect Base turningleftforless.com
Domain IPs 104.21.88.162, 172.67.186.9, 2606:4700:3033::6815:58a2, 2606:4700:3033::ac43:ba09
Redirect IPs 104.21.88.162, 172.67.186.9, 2606:4700:3033::6815:58a2, 2606:4700:3033::ac43:ba09
Response IP 172.67.186.9
Found Yes
Hash 2e28db75ac926902d66402247e40fa0a9919c17601d34326ecb7507fdb08ca70
SimHash a2030602f3d3

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /suggest/?*
Disallow /cdn-cgi/

Comments

  • This space intentionally left blank
  • If you want to learn about why our robots.txt looks like this, read this post: https://yoa.st/robots-txt