Log In Sign Up

branchen-info.net
robots.txt

Robots Exclusion Standard data for branchen-info.net

Resource Scan

Scan Details

Site Domain branchen-info.net
Base Domain branchen-info.net
Scan Status Ok
Last Scan2024-10-04T12:59:35+00:00
Next Scan 2024-10-11T12:59:35+00:00

Last Scan

Scanned2024-10-04T12:59:35+00:00
URL https://branchen-info.net/robots.txt
Redirect http://www.branchen-info.net/robots.txt
Redirect Domain www.branchen-info.net
Redirect Base branchen-info.net
Domain IPs 2a01:138:9002:2::4, 80.190.156.4
Redirect IPs 2a01:138:9002:2::4, 80.190.156.4
Response IP 80.190.156.4
Found Yes
Hash 3685a0cf038316f2c6a896bb2982d3d3060128487949f8935016a847ef38a183
SimHash 705cdf50c898

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

mbcrawler

Rule Path
Disallow /

censysinspect

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

*

Rule Path
Disallow /*/*.pdf
Disallow /m_branchen-info/
Disallow /bewertungs-tool/
Back to top

Comments

  • slow down Bingbot
  • Block Dotbot
  • https://moz.com/help/moz-procedures/crawlers/dotbot
  • Block Rogerbot
  • https://moz.com/help/moz-procedures/crawlers/rogerbot
  • Block Semrush
  • https://www.semrush.com/bot/
  • Block Seekport
  • https://bot.seekport.com/
  • Block Ahrefs
  • https://ahrefs.com/robot
  • Block SEOkicks
  • https://www.seokicks.de/robot.html
  • Block MegaIndex.ru
  • http://megaindex.ru/2.0
  • Block MJ12bot
  • https://mj12bot.com/
  • Block BLEXBot
  • http://webmeup-crawler.com/
  • Block CCBot
  • https://commoncrawl.org/big-picture/frequently-asked-questions/
  • Block VelenPublicWebCrawler
  • https://velen.io/
  • Block MBCrawler
  • https://monitorbacklinks.com
  • Block CensysInspect
  • https://about.censys.io/
  • Block GrapeshotCrawler
  • https://www.oracle.com/corporate/acquisitions/grapeshot/crawler.html
  • Block GPTBot
  • https://platform.openai.com/docs/gptbot
  • Block Google Bard, Vertex, etc.
  • https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers?hl=de
  • Block DataForSeoBot
  • https://dataforseo.com/dataforseo-bot
  • Block ImagesiftBot
  • https://imagesift.com/about
  • Block serpstatbot
  • https://serpstatbot.com/
Back to top