Log In Sign Up

autoit.dk
robots.txt

Robots Exclusion Standard data for autoit.dk

Resource Scan

Scan Details

Site Domain autoit.dk
Base Domain autoit.dk
Scan Status Ok
Last Scan2024-09-11T05:31:27+00:00
Next Scan 2024-10-11T05:31:27+00:00

Last Scan

Scanned2024-09-11T05:31:27+00:00
URL https://autoit.dk/robots.txt
Redirect https://www.autoit.dk/robots.txt
Redirect Domain www.autoit.dk
Redirect Base autoit.dk
Domain IPs 77.221.253.122
Redirect IPs 77.221.253.122
Response IP 77.221.253.122
Found Yes
Hash 8602c44c48362b426a96fae4720e79db59b74a4453bb656d97bb606688aaf725
SimHash 941672b42fee

Groups

serpstatbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

zoominfobot (zoominfobot at zoominfo dot com)

Rule Path
Disallow /

admantx

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

mozilla/3.0 (compatible; indy library)

Rule Path
Disallow /

indy library

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

softsearch.ru robot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-httplib2

Rule Path
Disallow /

expo9

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

yandex

Rule Path Comment
Disallow / blocks access to whole site for Yandex

eventmachine httpclient

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

nutch

Rule Path
Disallow /

maxpointcrawler/nutch-1.10

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

go-http-client/1.1

Rule Path
Disallow /

go 1.1 package http

Rule Path
Disallow /

wget

Rule Path
Disallow /

uipbot

Rule Path
Disallow /

uipbot/1.0 (uipbot@semasio.net)

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

free website submitter/1.0

Rule Path
Disallow /

free website submitter

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

domainstatsbot/1.0 (https://domainstats.com/pages/our-bot)

Rule Path
Disallow /
Back to top

Comments

  • Seekport Crawler
  • Block ZoominfoBot
  • Block ZoominfoBot
  • Block Admantx Bot
  • Block Heritrix
  • Block Indy Library Bot
  • Block Indy Library Bot
  • Block Mail.Ru
  • Block Majestic-12 Bot
  • Block Moz DotBot
  • Block Moz Rogerbot
  • Block PHPCrawl Bot
  • Block BLEXBot
  • Block Brandwatch's Magpie Crawler
  • Block SeznamBot
  • Block this bot
  • Block SoftSearch Bot
  • Block Uptime Bot
  • Buzzbot: http://www.buzzstream.com
Back to top