fredbortz.scienceblog.com
robots.txt

Robots Exclusion Standard data for fredbortz.scienceblog.com

Resource Scan

Scan Details

Site Domain fredbortz.scienceblog.com
Base Domain scienceblog.com
Scan Status Ok
Last Scan2025-03-08T01:46:26+00:00
Next Scan 2025-04-07T01:46:26+00:00

Last Scan

Scanned2025-03-08T01:46:26+00:00
URL https://fredbortz.scienceblog.com/robots.txt
Domain IPs 104.24.18.87, 104.24.19.87, 2606:4700:20::6818:1257, 2606:4700:20::6818:1357
Response IP 104.24.19.87
Found Yes
Hash b6ec2f86722800971dda891f34321ff310956b4ed981dc46cba7a5e08b331c29
SimHash 7b0e50d0c290

Groups

awariorssbot
awariosmartbot

Rule Path
Disallow /

*

Rule Path
Allow /wp-content/uploads/
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /admin/
Disallow /login/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /readme.html
Disallow /?author=*
Disallow *?s=*
Disallow *?p=*
Disallow */trackback/
Disallow */comments/
Disallow /*?*
Disallow /*.php$
Disallow /wp-content/debug.log

jetpack

Rule Path
Allow *

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

petalbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

Other Records

Field Value
sitemap /sitemap_index.xml

Comments

  • Only allow minimal required resources
  • Block WordPress system directories and files
  • Allow Jetpack (most Jetpack features will work through wp-json endpoint)
  • WordPress sitemap