Log In Sign Up

feedbuzzard.com
robots.txt

Robots Exclusion Standard data for feedbuzzard.com

Resource Scan

Scan Details

Site Domain feedbuzzard.com
Base Domain feedbuzzard.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-06-19T14:44:07+00:00
Next Scan 2025-07-03T14:44:07+00:00

Last Successful Scan

Scanned2025-06-04T14:43:13+00:00
URL https://feedbuzzard.com/robots.txt
Domain IPs 104.21.63.243, 172.67.173.94, 2606:4700:3031::6815:3ff3, 2606:4700:3035::ac43:ad5e
Response IP 104.21.63.243
Found Yes
Hash 83d56cbd447f8bd7f55b7ea2bc54d2359017249bf562cba73f0b534f1c753300
SimHash 245898310f32

Groups

googlebot

Rule Path
Allow *.js
Allow *.css
Allow /wp-admin/admin-ajax.php
Disallow /?s=
Disallow /search/
Disallow /wp-admin
Disallow /*/feed/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /trackback/

ahrefsbot

Rule Path
Disallow /$

ahrefssiteaudit

Rule Path
Disallow /$

*

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin
Disallow /wp-login.php
Disallow /trackback/
Disallow /wp-register.php
Back to top

Comments

  • Allow files critical for rendering
  • Allow AJAX - Do Not Remove
  • Prevent crawl-budget waste on search pages
  • Prevent private admin areas from being crawled
  • Prevent duplicate /feed/ pages from being crawled
  • Prevent login page crawls etc
  • Prevent register page crawls etc
  • Prevent Trackback Neg SEO
  • Add custom disallow rules for Google below
  • Rules for Ahrefs
  • Other user agents
  • Allow AJAX - Do Not Remove
Back to top