Log In Sign Up

compfence.se
robots.txt

Robots Exclusion Standard data for compfence.se

Resource Scan

Scan Details

Site Domain compfence.se
Base Domain compfence.se
Scan Status Ok
Last Scan2025-11-25T11:32:25+00:00
Next Scan 2025-12-25T11:32:25+00:00

Last Scan

Scanned2025-11-25T11:32:25+00:00
URL https://compfence.se/robots.txt
Redirect https://www.compfence.se/robots.txt
Redirect Domain www.compfence.se
Redirect Base compfence.se
Domain IPs 2a02:2350:5:10a:56:2203:3243:cb51, 46.30.213.10
Redirect IPs 185.158.133.1
Response IP 185.158.133.1
Found Yes
Hash 60cab9c543a16afbcf4bf182d0b11c6c730ee008a4c30a57847db8dc89e80fbc
SimHash e80cce01cdc7

Groups

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

*

Rule Path
Allow /
Disallow /api/
Disallow /*.json$
Disallow /*?*sort=
Disallow /*?*filter=

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.compfence.se/sitemap.xml
Back to top

Comments

  • Compfence Robots.txt
  • Optimized for search engine crawling
  • Sitemap location
  • Allow all major search engines full access
  • Social media bots
  • Allow all other bots
  • Disallow common non-public paths (if any exist in the future)
Back to top