Log In Sign Up

brandonag.org
robots.txt

Robots Exclusion Standard data for brandonag.org

Resource Scan

Scan Details

Site Domain brandonag.org
Base Domain brandonag.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-08-30T00:15:24+00:00
Next Scan 2025-09-29T00:15:24+00:00

Last Successful Scan

Scanned2025-07-08T23:43:21+00:00
URL https://brandonag.org/robots.txt
Redirect https://www.brandonag.org/robots.txt
Redirect Domain www.brandonag.org
Redirect Base brandonag.org
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Redirect IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.80.1
Found Yes
Hash 6cd0cf60345ce4f39095d389d51f4aae10c5e2150fa54c8d70d60b84f6b123ad
SimHash 011c8f16e7a9

Groups

*

Rule Path
Disallow /
Disallow /sitemap.xml

googlebot

Rule Path
Allow /
Allow /sitemap.xml

googlebot-image

Rule Path
Allow /
Allow /sitemap.xml

googlebot-mobile

Rule Path
Allow /
Allow /sitemap.xml

googlebot-news

Rule Path
Allow /
Allow /sitemap.xml

googlebot-video

Rule Path
Allow /
Allow /sitemap.xml

google-site-verification

Rule Path
Allow /
Allow /sitemap.xml

bingbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mozbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

sistrixcrawler

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

seolyzer

Rule Path
Disallow /

woorank

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

rankingcoachbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

botify

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

spyfu

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.brandonag.org/sitemap.xml
Back to top

Comments

  • Block all user agents by default
  • Allow only Googlebots to crawl the website and access the sitemap
  • Explicitly block other major search engines
  • Block all SEO tools and other crawlers
  • Block archive.org_bot (Wayback Machine)
  • Sitemap (accessible only to Googlebot)
Back to top