makeandtakes.com
robots.txt

Robots Exclusion Standard data for makeandtakes.com

Resource Scan

Scan Details

Site Domain makeandtakes.com
Base Domain makeandtakes.com
Scan Status Ok
Last Scan2024-06-14T18:25:47+00:00
Next Scan 2024-06-21T18:25:47+00:00

Last Scan

Scanned2024-06-14T18:25:47+00:00
URL https://makeandtakes.com/robots.txt
Domain IPs 104.21.13.86, 172.67.132.165, 2606:4700:3033::6815:d56, 2606:4700:3033::ac43:84a5
Response IP 172.67.132.165
Found Yes
Hash 3b9842361359dff64f234bb7dea5b2062751f0dbc6d6619b65130d8f6af9170f
SimHash 6008c8d2c7b1

Groups

scrapy

Rule Path
Allow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30