Log In Sign Up

hardelli.com
robots.txt

Robots Exclusion Standard data for hardelli.com

Resource Scan

Scan Details

Site Domain hardelli.com
Base Domain hardelli.com
Scan Status Ok
Last Scan2025-10-13T21:19:18+00:00
Next Scan 2025-10-20T21:19:18+00:00

Last Scan

Scanned2025-10-13T21:19:18+00:00
URL https://hardelli.com/robots.txt
Domain IPs 104.21.44.83, 172.67.197.228, 2606:4700:3030::ac43:c5e4, 2606:4700:3031::6815:2c53
Response IP 172.67.197.228
Found Yes
Hash 4041a90d2a42efe21abff090cda10a423687bfc7dc12efaeb97ac43fd8000b9b
SimHash 6c14d852eeb9

Groups

googlebot
googlebot-image
googlebot-video

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /*?replytocom
Disallow /wp-json/wp/v2/users
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Allow /wp-includes/js/
Allow /wp-includes/css/
Allow /wp-includes/images/

Other Records

Field Value
crawl-delay 0

bingbot
msnbot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /*?replytocom
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/

Other Records

Field Value
crawl-delay 1

yandex
duckduckbot
baiduspider
facebookexternalhit
twitterbot
linkedinbot
whatsapp
slackbot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Allow /wp-content/uploads/

Other Records

Field Value
crawl-delay 1

ahrefsbot
ahrefssiteaudit

Rule Path
Disallow /wp-*
Disallow /author/
Disallow /users/
Disallow /?s=
Disallow /search/

Other Records

Field Value
crawl-delay 10

semrushbot
semrushbot-sa
semrushbot-ba
semrushbot-bm

Rule Path
Disallow /wp-*
Disallow /author/
Disallow /?s=
Disallow /search/

Other Records

Field Value
crawl-delay 15

mj12bot

Rule Path
Disallow /wp-*
Disallow /author/
Disallow /category/
Disallow /tag/
Disallow /archive/
Disallow /?

Other Records

Field Value
crawl-delay 20

dotbot
rogerbot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/

Other Records

Field Value
crawl-delay 10

awariobot
blexbot
dataforseobot
domaincrawler
bytespider
aspiegelbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 30

gptbot
chatgpt-user
ccbot
anthropic-ai
claude-web
google-extended
perplexitybot
youbot
omgilibot

Rule Path
Disallow /

*

Rule Path
Disallow /wp-config.php
Disallow /wp-config-sample.php
Disallow /license.txt
Disallow /readme.html
Disallow /.htaccess
Disallow /.user.ini
Disallow /wp-settings.php
Disallow /wp-load.php
Disallow /wp-blog-header.php
Disallow /wp-cron.php
Disallow /wp-links-opml.php
Disallow /wp-activate.php
Disallow /xmlrpc.php
Disallow /.git/
Disallow /.svn/
Disallow /.hg/
Disallow /backup*/
Disallow /backups/
Disallow /cache/
Disallow /tmp/
Disallow /temp/
Disallow /logs/
Disallow /log/
Disallow /*.sql
Disallow /*.sql.gz
Disallow /*.log
Disallow /*.ini
Disallow /*.inc
Disallow /*.bak
Disallow /*.old
Disallow /*.save
Disallow /*.orig
Disallow /*.config
Disallow /*.conf
Disallow /*.env
Disallow /wp-content/uploads/*.php
Disallow /wp-content/uploads/wpforms/
Disallow /wp-content/uploads/gravity_forms/
Disallow /wp-content/uploads/ninja-forms/

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-content/themes/
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Allow /wp-includes/js/
Allow /wp-includes/css/
Allow /wp-includes/images/
Disallow /*?
Disallow /*?s=
Disallow /*?p=
Disallow /*?page_id=
Disallow /*?attachment_id=
Disallow /*?replytocom=
Disallow /trackback/
Disallow /feed/
Disallow /comments/
Disallow /author/
Disallow /category/*/page/
Disallow /tag/*/page/
Disallow /page/
Disallow /*utm_*%3D
Disallow /*fbclid%3D
Disallow /*gclid%3D
Disallow /*msclkid%3D

Other Records

Field Value
crawl-delay 2
Back to top

Comments

  • Master WordPress robots.txt - High-Security LEMP Stack
  • Version: 2.0 - Optimized for symlink deployment
  • Updated: 2025-06-30
  • Balances: Security | Performance | SEO | WordPress Protection
  • ==============================================================================
  • LEGITIMATE SEARCH ENGINES
  • ==============================================================================
  • Google - Full access with minimal restrictions
  • Bing/Microsoft
  • Other legitimate search engines
  • ==============================================================================
  • SEO TOOLS - Restricted Access
  • ==============================================================================
  • Ahrefs - Limited access
  • SEMrush - Limited access
  • Majestic - Heavily restricted
  • Moz - Moderate restrictions
  • ==============================================================================
  • AGGRESSIVE CRAWLERS - Significant Restrictions
  • ==============================================================================
  • ==============================================================================
  • AI/LLM CRAWLERS - Blocked by Default
  • ==============================================================================
  • ==============================================================================
  • SECURITY BLOCKS - All Bots
  • ==============================================================================
  • WordPress core files
  • Sensitive directories
  • Sensitive files by extension
  • WordPress uploads security
  • ==============================================================================
  • DEFAULT RULES - All Other User Agents
  • ==============================================================================
  • Core WordPress directories
  • Prevent duplicate content
  • Clean tracking parameters
  • Default crawl delay
  • Sitemap location (updated per site via symlink)
  • Sitemap: https://example.com/sitemap.xml
Back to top