Log In Sign Up

books.hostpath.de
robots.txt

Robots Exclusion Standard data for books.hostpath.de

Resource Scan

Scan Details

Site Domain books.hostpath.de
Base Domain hostpath.de
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-14T09:35:23+00:00
Next Scan 2024-12-13T09:35:23+00:00

Last Successful Scan

Scanned2022-11-28T17:43:20+00:00
URL https://books.hostpath.de/robots.txt
Response IP 78.46.83.245
Found Yes
Hash 819fa35ae86959651e7def34d26f15d87a7a708d0cd0162202811b749f0e31ba
SimHash 3a08198bf8e0

Groups

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /static/js/
Disallow /static/css/
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • AhrefsBot
  • see http://ahrefs.com/robot/
  • Disallow: /
  • SemrushBot
  • see http://www.semrush.com/bot.html
  • Disallow: /
  • To block SemrushBot from crawling your site for different SEO and technical issues:
  • To block SemrushBot from crawling your site for Backlink Audit tool:
  • To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SemrushBot from checking URLs on your site for SWA tool:
  • To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SemrushBot from crawling your site for Brand Monitoring:
  • To block SplitSignalBot from crawling your site for SplitSignal tool:
  • To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
  • DotBot
  • see https://opensiteexplorer.org/dotbot
  • Disallow: /
  • BLEXBot
  • see http://webmeup-crawler.com/
  • Disallow: /
  • MJ12bot
  • see http://mj12bot.com/
  • Disallow: /
  • PetalBot
  • see https://webmaster.petalsearch.com/site/petalbot
Back to top