Log In Sign Up

stuartmcmillen.com
robots.txt

Robots Exclusion Standard data for stuartmcmillen.com

Resource Scan

Scan Details

Site Domain stuartmcmillen.com
Base Domain stuartmcmillen.com
Scan Status Ok
Last Scan2024-06-21T13:27:55+00:00
Next Scan 2024-07-21T13:27:55+00:00

Last Scan

Scanned2024-06-21T13:27:55+00:00
URL https://stuartmcmillen.com/robots.txt
Domain IPs 104.21.25.247, 172.67.134.240, 2606:4700:3032::ac43:86f0, 2606:4700:3035::6815:19f7
Response IP 172.67.134.240
Found Yes
Hash 7999a605759180d782869aee631d75a9f62a38178118cfab556e519414e4239f
SimHash 9f67cc209dd0

Groups

*

Rule Path
Disallow /wp-json/

*

Rule Path
Disallow /wp-json/

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.stuartmcmillen.com/sitemap_index.xml
Back to top

Comments

  • This robots.txt file is manually created by Stuart, adapted from the Yoast website's file https://yoa.st/robots-txt
  • Global rules
  • -----------------
  • Sitemap
  • -----------------
  • Ban bots that don't benefit us.
  • --------------------------------
Back to top