Log In Sign Up

halle-hal.branchen-info.net
robots.txt

Robots Exclusion Standard data for halle-hal.branchen-info.net

Resource Scan

Scan Details

Site Domain halle-hal.branchen-info.net
Base Domain branchen-info.net
Scan Status Ok
Last Scan2024-09-23T11:14:31+00:00
Next Scan 2024-10-23T11:14:31+00:00

Last Scan

Scanned2024-09-23T11:14:31+00:00
URL https://halle-hal.branchen-info.net/robots.txt
Domain IPs 2a01:138:9002:2::4, 80.190.156.4
Response IP 80.190.156.4
Found Yes
Hash 462dd7742e19a4f36193ed959e2de8710de2679d10bf958e63b619641a5a4638
SimHash 325cd550c8d8

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

mbcrawler

Rule Path
Disallow /

censysinspect

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /
Back to top

Comments

  • slow down Bingbot
  • Block Dotbot
  • https://moz.com/help/moz-procedures/crawlers/dotbot
  • Block Rogerbot
  • https://moz.com/help/moz-procedures/crawlers/rogerbot
  • Block Semrush
  • https://www.semrush.com/bot/
  • Block Seekport
  • https://bot.seekport.com/
  • Block Ahrefs
  • https://ahrefs.com/robot
  • Block SEOkicks
  • https://www.seokicks.de/robot.html
  • Block MegaIndex.ru
  • http://megaindex.ru/2.0
  • Block MJ12bot
  • https://mj12bot.com/
  • Block BLEXBot
  • http://webmeup-crawler.com/
  • Block CCBot
  • https://commoncrawl.org/big-picture/frequently-asked-questions/
  • Block VelenPublicWebCrawler
  • https://velen.io/
  • Block MBCrawler
  • https://monitorbacklinks.com
  • Block CensysInspect
  • https://about.censys.io/
  • Block GrapeshotCrawler
  • https://www.oracle.com/corporate/acquisitions/grapeshot/crawler.html
  • Block DataForSeoBot
  • https://dataforseo.com/dataforseo-bot
  • Block GPTBot
  • https://platform.openai.com/docs/gptbot
  • Block Google Bard, Vertex, etc.
  • https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers?hl=de
Back to top