Log In Sign Up

pr3-articles.com
robots.txt

Robots Exclusion Standard data for pr3-articles.com

Resource Scan

Scan Details

Site Domain pr3-articles.com
Base Domain pr3-articles.com
Scan Status Ok
Last Scan2025-12-31T02:23:34+00:00
Next Scan 2026-01-30T02:23:34+00:00

Last Scan

Scanned2025-12-31T02:23:34+00:00
URL https://pr3-articles.com/robots.txt
Redirect http://www.pr3-articles.com/robots.txt
Redirect Domain www.pr3-articles.com
Redirect Base pr3-articles.com
Domain IPs 104.21.0.143, 172.67.128.19, 2606:4700:3034::ac43:8013, 2606:4700:3037::6815:8f
Redirect IPs 104.21.0.143, 172.67.128.19, 2606:4700:3034::ac43:8013, 2606:4700:3037::6815:8f
Response IP 172.67.128.19
Found Yes
Hash 98978fa4c0c194e7fd52ddf6a4990bf5315c80bcca0b8dab79023ef06567d238
SimHash 037e9749f7de

Groups

mediapartners-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

bingbot

Rule Path
Disallow

*

Rule Path
Allow /core/*.css
Allow /core/*.js
Allow /core/*.png
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.gif
Allow /modules/*.css
Allow /modules/*.js
Allow /modules/*.png
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.gif
Allow /themes/*.css
Allow /themes/*.js
Allow /themes/*.png
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.gif
Disallow /admin/
Disallow /user/
Disallow /node/add/
Disallow /node/preview/
Disallow /comment/reply/
Disallow /search/
Disallow /filter/tips/
Disallow /scripts/
Disallow /includes/
Disallow /profiles/
Disallow /update.php
Disallow /install.php
Disallow /xmlrpc.php
Disallow /*.txt$
Disallow /CHANGELOG.txt
Disallow /LICENSE.txt
Disallow /UPGRADE.txt
Disallow /INSTALL*

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • robots.txt for Drupal (Optimized)
  • Google and Ads bots - allow everything
  • Microsoft/Bing bots - allow everything
  • General user agents - set crawl rate and block sensitive paths
  • Allow access to CSS, JS, Images (Important for AdSense & Page Experience)
  • Block backend and sensitive directories
  • Block config and text files
  • Bot-specific crawl delays (these bots can overload)
Back to top