Log In Sign Up

anitajaeger.dk
robots.txt

Robots Exclusion Standard data for anitajaeger.dk

Resource Scan

Scan Details

Site Domain anitajaeger.dk
Base Domain anitajaeger.dk
Scan Status Ok
Last Scan2025-04-03T23:55:51+00:00
Next Scan 2025-05-03T23:55:51+00:00

Last Scan

Scanned2025-04-03T23:55:51+00:00
URL https://anitajaeger.dk/robots.txt
Domain IPs 52.17.45.100
Response IP 52.17.45.100
Found Yes
Hash 5c8baabde7fda16909409d95b1d866ae780ca545c37183f509d5c7a91b2b17d0
SimHash 7238f4f8e610

Groups

*

Rule Path
Disallow /request/*

morningscore bot

Rule Path
Disallow /request/*

Other Records

Field Value
crawl-delay 10

ccbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

siteliner

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

checkmarknetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Rule Path
Disallow /

checkmarknetwork/1.0

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

proximic

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

exabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

tracemyfile

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /
Back to top