Log In Sign Up

andersen-erhverv.dk
robots.txt

Robots Exclusion Standard data for andersen-erhverv.dk

Resource Scan

Scan Details

Site Domain andersen-erhverv.dk
Base Domain andersen-erhverv.dk
Scan Status Ok
Last Scan2025-05-15T04:11:45+00:00
Next Scan 2025-06-14T04:11:45+00:00

Last Scan

Scanned2025-05-15T04:11:45+00:00
URL https://andersen-erhverv.dk/robots.txt
Domain IPs 52.17.45.100
Response IP 52.17.45.100
Found Yes
Hash daed3545a81413a21ce36bfe679955448af0a45c5e49cd8e3f219bb457ffc26e
SimHash 73387544e220

Groups

ccbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

siteliner

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

checkmarknetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Rule Path
Disallow /

checkmarknetwork/1.0

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

proximic

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

tracemyfile

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

morningscore

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /
Back to top

Comments

  • Semrush
Back to top