freecrochettutorials.com
robots.txt

Robots Exclusion Standard data for freecrochettutorials.com

Resource Scan

Scan Details

Site Domain freecrochettutorials.com
Base Domain freecrochettutorials.com
Scan Status Ok
Last Scan2025-09-20T01:37:12+00:00
Next Scan 2025-09-27T01:37:12+00:00

Last Scan

Scanned2025-09-20T01:37:12+00:00
URL https://freecrochettutorials.com/robots.txt
Redirect https://www.freecrochettutorials.com/robots.txt
Redirect Domain www.freecrochettutorials.com
Redirect Base freecrochettutorials.com
Domain IPs 104.21.69.149, 172.67.209.93, 2606:4700:3033::6815:4595, 2606:4700:3035::ac43:d15d
Redirect IPs 104.18.37.69, 172.64.150.187, 2606:4700:4408::ac40:96bb, 2a06:98c1:3100::6812:2545
Response IP 104.18.37.69
Found Yes
Hash 92f7a00797c5ca5c6228408226229065c6efc699685ae1bec3477b082553f670
SimHash a4dffa70e613

Groups

*

Rule Path
Disallow /cdn-cgi/
Disallow /*add-to-cart%3D*

*

Rule Path
Disallow

gptbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.freecrochettutorials.com/post-sitemap.xml
sitemap https://www.freecrochettutorials.com/page-sitemap.xml
sitemap https://www.freecrochettutorials.com/tutorial-sitemap.xml
sitemap https://www.freecrochettutorials.com/category-sitemap.xml
sitemap https://www.freecrochettutorials.com/post_tag-sitemap.xml
sitemap https://www.freecrochettutorials.com/skill_level-sitemap.xml
sitemap https://www.freecrochettutorials.com/tutorial_type-sitemap.xml
sitemap https://www.freecrochettutorials.com/tutorial_feature-sitemap.xml
sitemap https://www.freecrochettutorials.com/author-sitemap.xml

Comments

  • Prevent Crawling Unnecessary Endpoints - Dynamically added by BigScoots