snarklesauce.com
robots.txt

Robots Exclusion Standard data for snarklesauce.com

Resource Scan

Scan Details

Site Domain snarklesauce.com
Base Domain snarklesauce.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-08T20:37:35+00:00
Next Scan 2026-01-06T20:37:35+00:00

Last Successful Scan

Scanned2025-06-11T12:17:30+00:00
URL https://snarklesauce.com/robots.txt
Domain IPs 103.106.229.82, 149.28.136.245, 2001:19f0:4400:2c6b:5400:5ff:fe0a:bcd8, 2401:c080:1400:555d:5400:4ff:fed2:82cf, 2a11:840:67:1b::348f:521f, 45.32.123.201
Response IP 149.28.136.245
Found Yes
Hash 6be7886227ef2558fda8fbf9be9ea55c16af842073f7c8bc1007d425e4443e11
SimHash 48184c02e6b2

Groups

*

Rule Path
Disallow /temp/
Disallow /admin/
Disallow /wp-admin/
Disallow /search/
Allow /blog/

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

Other Records

Field Value
sitemap https://snarklesauce.com/blog/sitemap.xml

Comments

  • Block common harmful bots