airliners.de
robots.txt

Robots Exclusion Standard data for airliners.de

Resource Scan

Scan Details

Site Domain airliners.de
Base Domain airliners.de
Scan Status Ok
Last Scan2024-05-23T20:10:38+00:00
Next Scan 2024-05-30T20:10:38+00:00

Last Scan

Scanned2024-05-23T20:10:38+00:00
URL https://airliners.de/robots.txt
Redirect https://www.airliners.de/robots.txt
Redirect Domain www.airliners.de
Redirect Base airliners.de
Domain IPs 134.119.0.225, 2a00:1158:5:e1::
Redirect IPs 134.119.0.225, 2a00:1158:5:e1::
Response IP 134.119.0.225
Found Yes
Hash f762aecab9081f7948701a34d04568359e08aa2df06025895276e2860cb0fa1b
SimHash 4836c0d240a0

Groups

*

Rule Path
Allow /
Disallow /karriere/r/
Disallow /nova/
Disallow /admin/
Disallow /backend/
Disallow /vendor/
Disallow /settings/*
Disallow /page-cache/*
Disallow /firmen/*/visit/*
Disallow /*?*refinementList
Disallow /suche

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sirdatabot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

grapeshotcrawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3