Log In Sign Up

bluediamondattachments.com
robots.txt

Robots Exclusion Standard data for bluediamondattachments.com

Resource Scan

Scan Details

Site Domain bluediamondattachments.com
Base Domain bluediamondattachments.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-12T18:46:35+00:00
Next Scan 2026-01-10T18:46:35+00:00

Last Successful Scan

Scanned2025-05-22T17:29:01+00:00
URL https://bluediamondattachments.com/robots.txt
Redirect https://www.bluediamondattachments.com/robots.txt
Redirect Domain www.bluediamondattachments.com
Redirect Base bluediamondattachments.com
Domain IPs 104.26.14.100, 104.26.15.100, 172.67.70.156, 2606:4700:20::681a:e64, 2606:4700:20::681a:f64, 2606:4700:20::ac43:469c
Redirect IPs 104.26.14.100, 104.26.15.100, 172.67.70.156, 2606:4700:20::681a:e64, 2606:4700:20::681a:f64, 2606:4700:20::ac43:469c
Response IP 104.26.15.100
Found Yes
Hash c1362812c7c7f5dabc4650b1a2cc4d4fb0639c67cbf649766c09311dba2c0ff8
SimHash 62e63d6084bc

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/
Disallow /search/

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

psbot

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

baiduspider/2.0;+http://www.baidu.com/search/spider.html

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

mozilla/5.0(compatible; baiduspider/2.0;+http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandex.com

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sebot-wa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

sitecheckerbotcrawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

scrapy

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3
Back to top

Other Records

Field Value
sitemap https://www.bluediamondattachments.com/sitemaps-1-sitemap.xml
sitemap https://dealers.bluediamondattachments.com/sitemaps-1-sitemap.xml
Back to top

Comments

  • live - don't allow web crawlers to index cpresources/ or vendor/
  • even good bots don't need to crawl the site with random search queries -Stuart
  • Disallow ChatGPT bot, as there's no benefit to allowing it to index your site
  • Disallow Google Bard and Vertex AI bots, as there's no benefit to allowing it to index your site
  • Baiduspider
  • Yandex
  • Block BlexBot
  • Block MegaIndex.ru
  • Block Ahrefs
  • Block MJ12bot as it is just noise
  • -stuart
  • Block Go-http-client as it's causing resource issues
  • Block Bytespider
  • Slow down SEBot-WA. Hard block next time it behaves badly
  • Slow down SiteCheckerPro, crawled cached pages at very high rate
  • Slow down Scrapy, crawled whole site high rate multiple days in a row
Back to top

Warnings

  • 6 invalid lines.
Back to top