Log In Sign Up

zwerfkei.nl
robots.txt

Robots Exclusion Standard data for zwerfkei.nl

Resource Scan

Scan Details

Site Domain zwerfkei.nl
Base Domain zwerfkei.nl
Scan Status Ok
Last Scan2024-09-25T13:20:38+00:00
Next Scan 2024-10-25T13:20:38+00:00

Last Scan

Scanned2024-09-25T13:20:38+00:00
URL https://zwerfkei.nl/robots.txt
Redirect https://www.zwerfkei.nl/robots.txt
Redirect Domain www.zwerfkei.nl
Redirect Base zwerfkei.nl
Domain IPs 87.233.246.166
Redirect IPs 87.233.246.166
Response IP 87.233.246.166
Found Yes
Hash 6edc299bacf7f5ee54ea6b7f754542c6d834dde97599b9e3598b35ba01831396
SimHash 480751634c1b

Groups

panscient.com

Rule Path
Disallow /

vscooter

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

yandex

Rule Path
Disallow /

taptubot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

twengabot/1.1

Rule Path
Disallow /

twengabot/2.0

Rule Path
Disallow /

twenga*

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

infopath

Rule Path
Disallow /

infopath.2

Rule Path
Disallow /

swebot

Rule Path
Disallow /

ec2linkfinder

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

searchmetericsbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

ip-web-crawler.com

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

aboundex

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

spbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

riddler

Rule Path
Disallow /

loadtimebot

Rule Path
Disallow /

obot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

advbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

lssrocketcrawler

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

nutch

Rule Path
Disallow /

tbot-nutch

Rule Path
Disallow /

thunderstone

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

ranksonicbot

Rule Path
Disallow /

betabot

Rule Path
Disallow /

parsijoo-bot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

gocrawl

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

applebot

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

*

Rule Path
Disallow /*.PDF
Disallow /*.pdf
Disallow /*.mp3
Disallow /*.MP3
Disallow /*.MOV
Disallow /*.mov
Disallow /*.AVI
Disallow /*.avi
Disallow /system
Disallow /backend
Disallow /touch/
Disallow /ajax/
Disallow /api/
Disallow /crontab
Disallow /crossdomain
Disallow /klant
Disallow /reklamezuil
Disallow /info_display
Disallow /download
Disallow /zoeken
Disallow /articles/stockstatus*
Disallow /reviews/review_schrijven*
Allow /
Back to top

Other Records

Field Value
sitemap https://www.zwerfkei.nl/sitemap.xml
Back to top

Comments

  • baidu
  • User-agent: Baiduspider
  • Disallow: /
  • User-agent: archive.org_bot
  • Disallow: /
  • yahoo
  • User-agent: Slurp
  • Disallow: /
  • part of bing
  • User-agent: msnbot
  • Disallow: /
Back to top