Log In Sign Up

pr6-articles.com
robots.txt

Robots Exclusion Standard data for pr6-articles.com

Resource Scan

Scan Details

Site Domain pr6-articles.com
Base Domain pr6-articles.com
Scan Status Ok
Last Scan2025-12-18T09:16:11+00:00
Next Scan 2026-01-17T09:16:11+00:00

Last Scan

Scanned2025-12-18T09:16:11+00:00
URL https://www.pr6-articles.com/robots.txt
Domain IPs 104.21.56.248, 172.67.157.58, 2606:4700:3034::6815:38f8, 2606:4700:3035::ac43:9d3a
Response IP 172.67.157.58
Found Yes
Hash 98978fa4c0c194e7fd52ddf6a4990bf5315c80bcca0b8dab79023ef06567d238
SimHash 037e9749f7de

Groups

mediapartners-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

bingbot

Rule Path
Disallow

*

Rule Path
Allow /core/*.css
Allow /core/*.js
Allow /core/*.png
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.gif
Allow /modules/*.css
Allow /modules/*.js
Allow /modules/*.png
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.gif
Allow /themes/*.css
Allow /themes/*.js
Allow /themes/*.png
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.gif
Disallow /admin/
Disallow /user/
Disallow /node/add/
Disallow /node/preview/
Disallow /comment/reply/
Disallow /search/
Disallow /filter/tips/
Disallow /scripts/
Disallow /includes/
Disallow /profiles/
Disallow /update.php
Disallow /install.php
Disallow /xmlrpc.php
Disallow /*.txt$
Disallow /CHANGELOG.txt
Disallow /LICENSE.txt
Disallow /UPGRADE.txt
Disallow /INSTALL*

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • robots.txt for Drupal (Optimized)
  • Google and Ads bots - allow everything
  • Microsoft/Bing bots - allow everything
  • General user agents - set crawl rate and block sensitive paths
  • Allow access to CSS, JS, Images (Important for AdSense & Page Experience)
  • Block backend and sensitive directories
  • Block config and text files
  • Bot-specific crawl delays (these bots can overload)
Back to top