Log In Sign Up

peterfisk.com
robots.txt

Robots Exclusion Standard data for peterfisk.com

Resource Scan

Scan Details

Site Domain peterfisk.com
Base Domain peterfisk.com
Scan Status Ok
Last Scan2025-10-04T15:51:00+00:00
Next Scan 2025-11-03T15:51:00+00:00

Last Scan

Scanned2025-10-04T15:51:00+00:00
URL https://peterfisk.com/robots.txt
Domain IPs 162.159.135.42
Response IP 162.159.135.42
Found Yes
Hash 28096d23f82a547d37592440aa493e005c98b7e96f03b373ca4ae456a63d89f6
SimHash 583899d2d3b0

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow /wp-admin/

gptbot
ccbot
chatgpt-user

Rule Path
Disallow /

semrushbot
ahrefsbot
petalbot
mj12bot
dotbot
mauibot

Rule Path
Disallow /

googlebot
*

Rule Path
Disallow
Back to top

Comments

  • START BOOKSWARM ROBOTS.TXT TEMPLATE
  • -----------------------------------
  • Set the crawl delay to 5 seconds - not all search engines will honour this
  • Tell all user agents to ignore wp-admin
  • Block AI bots
  • Block these other bots
  • Allow Googlebot and other bots not specific above
  • ---------------------------------
  • END BOOKSWARM ROBOTS.TXT TEMPLATE
  • _
  • [ ]
  • ( )
  • |>|
  • __/===\__
  • //| o=o |\\
  • <] | o=o | [>
  • \=====/
  • / / | \ \
  • <_________>
Back to top