freistaetter-feldbahn.de
robots.txt

Robots Exclusion Standard data for freistaetter-feldbahn.de

Resource Scan

Scan Details

Site Domain freistaetter-feldbahn.de
Base Domain freistaetter-feldbahn.de
Scan Status Ok
Last Scan2024-09-17T19:36:10+00:00
Next Scan 2024-10-17T19:36:10+00:00

Last Scan

Scanned2024-09-17T19:36:10+00:00
URL http://freistaetter-feldbahn.de/robots.txt
Domain IPs 178.20.102.48
Response IP 178.20.102.48
Found Yes
Hash 911b76e8aad5936cd1c46f670f4726d075f0c34224820248a4b4470fe3591dde
SimHash 1b38476a7a89

Groups

*

Rule Path
Disallow /administrator/
Disallow /cache/
Disallow /cgi-bin/
Disallow /cli/
Disallow /components/
Disallow /images/
Disallow /includes/
Disallow /language/
Disallow /libraries/
Disallow /logs/
Disallow /media/
Disallow /modules/
Disallow /plugins/
Disallow /templates/
Disallow /tmp/

abonti

Rule Path
Disallow /

acebookexternalhit

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

bot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cpython

Rule Path
Disallow /

curl

Rule Path
Disallow /

dbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

domnutch-bot

Rule Path
Disallow /

domnutch

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

finbot

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

fetcher1-5

Rule Path
Disallow /

fetcher2

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

genieo

Rule Path
Disallow /

ggot

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

hbtools

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

inagist url resolver

Rule Path
Disallow /

jakarta commons-httpclient

Rule Path
Disallow /

java

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mediabot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

mixbot

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mnogosearch

Rule Path
Disallow /

monobot

Rule Path
Disallow /

nbot

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

nutch

Rule Path
Disallow /

nutchcvs

Rule Path
Disallow /

obot

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

pixray-seeker

Rule Path
Disallow /

proximic

Rule Path
Disallow /

publiclibraryarchive.org

Rule Path
Disallow /

python

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

pythonurllib

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semalt.com

Rule Path
Disallow /

semalt.semalt.com

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spider02

Rule Path
Disallow /

surveybot_ignoreip

Rule Path
Disallow /

synapse

Rule Path
Disallow /

tarantula

Rule Path
Disallow /

tweetedtimes bot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

u2bot

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webfilter

Rule Path
Disallow /

webindex

Rule Path
Disallow /

webmasteraid

Rule Path
Disallow /

webtarantula

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

xenu's link sleuth

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 4

yandexbot

Rule Path
Disallow /

Warnings

  • 2 invalid lines.