Log In Sign Up

paddlemethod.com
robots.txt

Robots Exclusion Standard data for paddlemethod.com

Resource Scan

Scan Details

Site Domain paddlemethod.com
Base Domain paddlemethod.com
Scan Status Ok
Last Scan2025-10-22T16:24:49+00:00
Next Scan 2025-10-29T16:24:49+00:00

Last Scan

Scanned2025-10-22T16:24:49+00:00
URL https://paddlemethod.com/robots.txt
Domain IPs 104.238.152.141
Response IP 104.238.152.141
Found Yes
Hash 13918874ec526781a93c7f4a0484a951db0a80546b40e98fe7ea2db7acda16d0
SimHash 81354a216583

Groups

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-news

Rule Path
Disallow

google-other

Rule Path
Disallow

bingbot

Rule Path
Disallow

slurp

Product Comment
slurp Yahoo (powered by Bing, minor but harmless)
Rule Path
Disallow

applebot

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

gptbot

Rule Path
Disallow

ccbot

Product Comment
ccbot Common Crawl – used by many AI models
Rule Path
Disallow

perplexitybot

Rule Path
Disallow

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

wget

Rule Path
Disallow /

curl

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

*

Rule Path
Disallow /cgi-bin/
Disallow /tmp/
Disallow /private/
Disallow /wp-content/uploads/wc-logs/
Disallow /wp-content/uploads/woocommerce_transient_files/
Disallow /wp-content/uploads/woocommerce_uploads/
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://paddlemethod.com/sitemap_index.xml
Back to top

Comments

  • ==============================
  • Robots.txt – Smart Allow/Block Policy
  • ==============================
  • --- Core Search Engines (Always Allow) ---
  • --- Strategic AI/LLM Crawlers (Allow for Future-Proofing) ---
  • --- Regional Search Engines (Allow if relevant, else Block) ---
  • --- SEO Tools (Block unless you actively use them) ---
  • --- Junk/Generic Scrapers (Always Block) ---
  • START YOAST BLOCK
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
Back to top