Log In Sign Up

teoritjek.dk
robots.txt

Robots Exclusion Standard data for teoritjek.dk

Resource Scan

Scan Details

Site Domain teoritjek.dk
Base Domain teoritjek.dk
Scan Status Ok
Last Scan2026-03-06T10:47:14+00:00
Next Scan 2026-03-13T10:47:14+00:00

Last Scan

Scanned2026-03-06T10:47:14+00:00
URL https://teoritjek.dk/robots.txt
Domain IPs 104.21.20.99, 172.67.192.17, 2606:4700:3031::ac43:c011, 2606:4700:3032::6815:1463
Response IP 172.67.192.17
Found Yes
Hash f0e91ce5c8f48e87d3a270b2fa85cf3d501c6b119528ddf9d7964fdbebcd2475
SimHash 5c5d5c50e792

Groups

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 2

bingbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 2

slurp

Rule Path
Allow /
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 2

*

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/
Disallow /korekort-pris-beregner/data

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://teoritjek.dk/sitemap.xml
Back to top

Comments

  • TeoriTjek Robots.txt
  • We invest significant resources in data analysis, enrichment, and comparison tools.
  • Unauthorized automated data collection is prohibited.
  • For legitimate data licensing inquiries, contact us through our website.
  • Block aggressive scrapers and crawlers
  • Block AI training crawlers
  • Allow legitimate search engines with rate limiting
  • Default rules for all other bots
  • Sitemap
Back to top