Log In Sign Up

books.infosec.exchange
robots.txt

Robots Exclusion Standard data for books.infosec.exchange

Resource Scan

Scan Details

Site Domain books.infosec.exchange
Base Domain infosec.exchange
Scan Status Ok
Last Scan2024-09-29T10:21:05+00:00
Next Scan 2024-09-30T10:21:05+00:00

Last Scan

Scanned2024-09-29T10:21:05+00:00
URL https://books.infosec.exchange/robots.txt
Domain IPs 2a01:4f8:2201:3347::2, 46.4.254.144
Response IP 46.4.254.144
Found Yes
Hash dc03ecdfdc54f290cfb4f72e4190c4ed41a9fb0d8d6f609ea871e6e87a7d4dd4
SimHash ba88198bf8e0

Groups

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

*

Rule Path
Disallow /static/js/
Disallow /static/css/

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • AhrefsBot
  • see http://ahrefs.com/robot/
  • Disallow: /
  • SemrushBot
  • see http://www.semrush.com/bot.html
  • Disallow: /
  • To block SemrushBot from crawling your site for different SEO and technical issues:
  • To block SemrushBot from crawling your site for Backlink Audit tool:
  • To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SemrushBot from checking URLs on your site for SWA tool:
  • To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SemrushBot from crawling your site for Brand Monitoring:
  • To block SplitSignalBot from crawling your site for SplitSignal tool:
  • To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
  • DotBot
  • see https://opensiteexplorer.org/dotbot
  • Disallow: /
  • BLEXBot
  • see http://webmeup-crawler.com/
  • Disallow: /
  • MJ12bot
  • see http://mj12bot.com/
  • Disallow: /
  • PetalBot
  • see https://webmaster.petalsearch.com/site/petalbot
Back to top