Log In Sign Up

vetline.de
robots.txt

Robots Exclusion Standard data for vetline.de

Resource Scan

Scan Details

Site Domain vetline.de
Base Domain vetline.de
Scan Status Ok
Last Scan2024-09-22T08:40:07+00:00
Next Scan 2024-10-22T08:40:07+00:00

Last Scan

Scanned2024-09-22T08:40:07+00:00
URL https://vetline.de/robots.txt
Redirect https://www.vetline.de/robots.txt
Redirect Domain www.vetline.de
Redirect Base vetline.de
Domain IPs 35.242.229.239, 35.246.184.45, 35.246.248.138
Redirect IPs 35.246.184.45
Response IP 35.242.229.239
Found Yes
Hash 104eb5ba300d787c13e8668ba4715067f7239619675a7890e3446a1c073b1828
SimHash 39943f08aca8

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Disallow /core/
Disallow /profiles/
Disallow /README.md
Disallow /composer/Metapackage/README.txt
Disallow /composer/Plugin/ProjectMessage/README.md
Disallow /composer/Plugin/Scaffold/README.md
Disallow /composer/Plugin/VendorHardening/README.txt
Disallow /composer/Template/README.txt
Disallow /modules/README.txt
Disallow /sites/README.txt
Disallow /themes/README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /media/oembed
Disallow /*/media/oembed
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password
Disallow /index.php/user/register
Disallow /index.php/user/login
Disallow /index.php/user/logout
Disallow /index.php/media/oembed
Disallow /index.php/*/media/oembed
Disallow /reifentests*
Disallow /verzeichnis/suche/
Disallow /index.php/verzeichnis/suche/
Disallow /stellenmarkt/

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

semantic-vision.com

Rule Path
Disallow /

testcrawler

Rule Path
Disallow /

denic-crawler/42

Rule Path
Disallow /

contao/crawler

Rule Path
Disallow /

bingbot

Rule Path
Disallow /*send_as_pdf*
Disallow /*download_as_pdf*
Disallow /*node*
Disallow /*taxonomy*

adstxtcrawlertp

Rule Path
Disallow /

adstxtcrawler

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

brightedge

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

clubhouse-linkpreview

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

criteobot

Rule Path
Disallow /

contao

Rule Path
Disallow /

crawler_eb_germany_2.0

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ev-crawler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

lcc

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

lightspeedsystemscrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mbcrawler

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

netvibes

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

obot

Rule Path
Disallow /

ogscrper

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

pmgcrawler

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pubmatic

Rule Path
Disallow /

radian6

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

sitepark

Rule Path
Disallow /

testcrawler

Rule Path
Disallow /

tineye-bot-live

Rule Path
Disallow /

x11

Rule Path
Disallow /

yandex

Rule Path
Disallow /
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Legal notice: vetline.de expressly reserves the right to use its content for commercial text and data mining (§ 44b UrhG).
  • The use of robots or other automated means to access vetline.de and/or all its subpages or collect or mine data from vetline.de and/or all its subpages without the express permission of vetline.de is strictly prohibited.
  • If you would like to apply for permission to crawl vetline.de and/or all its subpages, collect or use data, please contact lizenzen@schluetersche.de
  • -------------------------------------
  • Extra-Robots
  • Slurp
  • X11; compatible; semantic-visions.com crawler
  • Testcrawler
  • DENIC Crawler
  • contao/crawler Crawler
  • Bing
  • AdsTxtCrawlerTP
  • AdsTxtCrawler
  • arquivo-web-crawler
  • barkrowler
  • BLEXBot
  • BrightEdge
  • ClaudeBot
  • Clubhouse-LinkPreview
  • CriteoBot
  • contao
  • crawler_eb_germany_2.0
  • Dotbot
  • ev-crawler
  • Exabot
  • GrapeshotCrawler
  • ias_crawler
  • IonCrawl
  • LCC
  • LivelapBot
  • LightspeedSystemsCrawler
  • magpie-crawler
  • MBCrawler
  • MegaIndex.ru
  • metajobbot
  • MixrankBot
  • MJ12bot
  • Netvibes
  • Nicecrawler
  • oBot
  • OgScrper
  • PerplexityBot
  • PetaBot
  • PMGCrawler
  • proximic
  • PubMatic
  • radian6
  • R6_CommentReader
  • R6_FeedFetcher
  • Scrapy
  • SemanticScholarBot
  • SEOkicks
  • SemrushBot
  • SEOkicks
  • sitepark
  • Testcrawler
  • TinEye-bot-live
  • X11
  • Yandex
Back to top