Log In Sign Up

lire.boitam.eu
robots.txt

Robots Exclusion Standard data for lire.boitam.eu

Resource Scan

Scan Details

Site Domain lire.boitam.eu
Base Domain boitam.eu
Scan Status Ok
Last Scan2025-07-22T07:27:03+00:00
Next Scan 2025-07-23T07:27:03+00:00

Last Scan

Scanned2025-07-22T07:27:03+00:00
URL https://lire.boitam.eu/robots.txt
Domain IPs 2001:bc8:1200:d:ec4:7aff:fe71:c2aa, 51.159.70.141
Response IP 51.159.70.141
Found Yes
Hash dc03ecdfdc54f290cfb4f72e4190c4ed41a9fb0d8d6f609ea871e6e87a7d4dd4
SimHash ba88198bf8e0

Groups

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

*

Rule Path
Disallow /static/js/
Disallow /static/css/

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • AhrefsBot
  • see http://ahrefs.com/robot/
  • Disallow: /
  • SemrushBot
  • see http://www.semrush.com/bot.html
  • Disallow: /
  • To block SemrushBot from crawling your site for different SEO and technical issues:
  • To block SemrushBot from crawling your site for Backlink Audit tool:
  • To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SemrushBot from checking URLs on your site for SWA tool:
  • To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SemrushBot from crawling your site for Brand Monitoring:
  • To block SplitSignalBot from crawling your site for SplitSignal tool:
  • To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
  • DotBot
  • see https://opensiteexplorer.org/dotbot
  • Disallow: /
  • BLEXBot
  • see http://webmeup-crawler.com/
  • Disallow: /
  • MJ12bot
  • see http://mj12bot.com/
  • Disallow: /
  • PetalBot
  • see https://webmaster.petalsearch.com/site/petalbot
Back to top