swbf2stats.net
robots.txt

Robots Exclusion Standard data for swbf2stats.net

Resource Scan

Scan Details

Site Domain swbf2stats.net
Base Domain swbf2stats.net
Scan Status Ok
Last Scan2026-02-25T14:59:57+00:00
Next Scan 2026-03-04T14:59:57+00:00

Last Scan

Scanned2026-02-25T14:59:57+00:00
URL https://swbf2stats.net/robots.txt
Domain IPs 108.175.12.184
Response IP 108.175.12.184
Found Yes
Hash 61c1cf3497e1f9ae9032921cb1a6ba8a70f4aa8fa7da3faf98a99b3d3843281c
SimHash a8215d92a722

Groups

*

Rule Path
Disallow /blackhole/
Disallow /logs/
Disallow /private/
Disallow /vendor/
Disallow /site_cache/
Disallow /errors/
Disallow /submit
Disallow /search_results

Other Records

Field Value
sitemap https://swbf2stats.net/sitemap.xml

Comments

  • Global rules for all crawlers
  • Do not crawl trap or utility folders
  • Do not crawl error handlers
  • Optional: prevent crawling of internal utility endpoints
  • Sitemap