Log In Sign Up

jothomasauthor.com
robots.txt

Robots Exclusion Standard data for jothomasauthor.com

Resource Scan

Scan Details

Site Domain jothomasauthor.com
Base Domain jothomasauthor.com
Scan Status Ok
Last Scan2025-10-02T03:54:47+00:00
Next Scan 2025-11-01T03:54:47+00:00

Last Scan

Scanned2025-10-02T03:54:47+00:00
URL https://jothomasauthor.com/robots.txt
Domain IPs 104.21.48.6, 172.67.175.40, 2606:4700:3030::ac43:af28, 2606:4700:3034::6815:3006
Response IP 104.21.48.6
Found Yes
Hash 28096d23f82a547d37592440aa493e005c98b7e96f03b373ca4ae456a63d89f6
SimHash 583899d2d3b0

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow /wp-admin/

gptbot
ccbot
chatgpt-user

Rule Path
Disallow /

semrushbot
ahrefsbot
petalbot
mj12bot
dotbot
mauibot

Rule Path
Disallow /

googlebot
*

Rule Path
Disallow
Back to top

Comments

  • START BOOKSWARM ROBOTS.TXT TEMPLATE
  • -----------------------------------
  • Set the crawl delay to 5 seconds - not all search engines will honour this
  • Tell all user agents to ignore wp-admin
  • Block AI bots
  • Block these other bots
  • Allow Googlebot and other bots not specific above
  • ---------------------------------
  • END BOOKSWARM ROBOTS.TXT TEMPLATE
  • _
  • [ ]
  • ( )
  • |>|
  • __/===\__
  • //| o=o |\\
  • <] | o=o | [>
  • \=====/
  • / / | \ \
  • <_________>
Back to top