Log In Sign Up

slbasics.com
robots.txt

Robots Exclusion Standard data for slbasics.com

Resource Scan

Scan Details

Site Domain slbasics.com
Base Domain slbasics.com
Scan Status Ok
Last Scan2025-08-06T05:07:01+00:00
Next Scan 2025-08-20T05:07:01+00:00

Last Scan

Scanned2025-08-06T05:07:01+00:00
URL https://slbasics.com/robots.txt
Domain IPs 23.227.38.65
Response IP 23.227.38.65
Found Yes
Hash c52d47c1e41ad8f110172a549734bc996d8c67951fe4e459c07469b12595df18
SimHash 245a0e808fd3

Groups

*

Rule Path
Allow /

httrack

Rule Path
Disallow /

wget

Rule Path
Disallow /

curl

Rule Path
Disallow /

libwww

Rule Path
Disallow /

scraper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • robots.txt for SL Basics – Default + Anti-Scraper Rules
  • Allow all crawlers access to everything by default
  • Block known website cloners and scrapers
  • Shopify-specific defaults
Back to top