stingynomads.com
robots.txt

Robots Exclusion Standard data for stingynomads.com

Resource Scan

Scan Details

Site Domain stingynomads.com
Base Domain stingynomads.com
Scan Status Ok
Last Scan6/7/2025, 5:44:52 PM
Next Scan 6/14/2025, 5:44:52 PM

Last Scan

Scanned6/7/2025, 5:44:52 PM
URL https://stingynomads.com/robots.txt
Domain IPs 104.21.48.198, 172.67.187.233, 2606:4700:3034::ac43:bbe9, 2606:4700:3037::6815:30c6
Response IP 104.21.48.198
Found Yes
Hash 5e80af117765778eff580a39de885b43656c814fe73256d72949ea0cb255ae38
SimHash a4404b10e610

Groups

*

Rule Path
Disallow /cdn-cgi/
Disallow /*add-to-cart%3D*

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Comments

  • Prevent Crawling Unnecessary Endpoints - Dynamically added by BigScoots