Log In Sign Up

triobiler.dk
robots.txt

Robots Exclusion Standard data for triobiler.dk

Resource Scan

Scan Details

Site Domain triobiler.dk
Base Domain triobiler.dk
Scan Status Ok
Last Scan2025-12-23T18:50:54+00:00
Next Scan 2026-01-22T18:50:54+00:00

Last Scan

Scanned2025-12-23T18:50:54+00:00
URL https://triobiler.dk/robots.txt
Redirect https://www.triobiler.dk/robots.txt
Redirect Domain www.triobiler.dk
Redirect Base triobiler.dk
Domain IPs 77.221.253.122
Redirect IPs 77.221.253.122
Response IP 77.221.253.122
Found Yes
Hash e5a84f4b1ee4854bd076bc0887bafa6bb076ba4607a6fa5ce06256c9e4d3c108
SimHash 141672b66fee

Groups

scrapy

Rule Path
Allow /

serpstatbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

zoominfobot (zoominfobot at zoominfo dot com)

Rule Path
Disallow /

admantx

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

mozilla/3.0 (compatible; indy library)

Rule Path
Disallow /

indy library

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

softsearch.ru robot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-httplib2

Rule Path
Disallow /

expo9

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

yandex

Rule Path Comment
Disallow / blocks access to whole site for Yandex

eventmachine httpclient

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

nutch

Rule Path
Disallow /

maxpointcrawler/nutch-1.10

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

go-http-client/1.1

Rule Path
Disallow /

go 1.1 package http

Rule Path
Disallow /

wget

Rule Path
Disallow /

uipbot

Rule Path
Disallow /

uipbot/1.0 (uipbot@semasio.net)

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

free website submitter/1.0

Rule Path
Disallow /

free website submitter

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

domainstatsbot/1.0 (https://domainstats.com/pages/our-bot)

Rule Path
Disallow /
Back to top

Comments

  • Seekport Crawler
  • Block ZoominfoBot
  • Block ZoominfoBot
  • Block Admantx Bot
  • Block Heritrix
  • Block Indy Library Bot
  • Block Indy Library Bot
  • Block Mail.Ru
  • Block Majestic-12 Bot
  • Block Moz DotBot
  • Block Moz Rogerbot
  • Block PHPCrawl Bot
  • Block BLEXBot
  • Block Brandwatch's Magpie Crawler
  • Block SeznamBot
  • Block this bot
  • Block SoftSearch Bot
  • Block Uptime Bot
  • Buzzbot: http://www.buzzstream.com
Back to top