Log In Sign Up

lit.rickosborne.org
robots.txt

Robots Exclusion Standard data for lit.rickosborne.org

Resource Scan

Scan Details

Site Domain lit.rickosborne.org
Base Domain rickosborne.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-06-01T02:02:19+00:00
Next Scan 2025-08-30T02:02:19+00:00

Last Successful Scan

Scanned2023-08-18T01:54:59+00:00
URL https://lit.rickosborne.org/robots.txt
Domain IPs 204.48.54.150
Response IP 204.48.54.150
Found Yes
Hash dc03ecdfdc54f290cfb4f72e4190c4ed41a9fb0d8d6f609ea871e6e87a7d4dd4
SimHash ba88198bf8e0

Groups

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

*

Rule Path
Disallow /static/js/
Disallow /static/css/

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • AhrefsBot
  • see http://ahrefs.com/robot/
  • Disallow: /
  • SemrushBot
  • see http://www.semrush.com/bot.html
  • Disallow: /
  • To block SemrushBot from crawling your site for different SEO and technical issues:
  • To block SemrushBot from crawling your site for Backlink Audit tool:
  • To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SemrushBot from checking URLs on your site for SWA tool:
  • To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SemrushBot from crawling your site for Brand Monitoring:
  • To block SplitSignalBot from crawling your site for SplitSignal tool:
  • To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
  • DotBot
  • see https://opensiteexplorer.org/dotbot
  • Disallow: /
  • BLEXBot
  • see http://webmeup-crawler.com/
  • Disallow: /
  • MJ12bot
  • see http://mj12bot.com/
  • Disallow: /
  • PetalBot
  • see https://webmaster.petalsearch.com/site/petalbot
Back to top