security.org
robots.txt

Robots Exclusion Standard data for security.org

Resource Scan

Scan Details

Site Domain security.org
Base Domain security.org
Scan Status Ok
Last Scan2025-12-15T17:47:16+00:00
Next Scan 2025-12-29T17:47:16+00:00

Last Scan

Scanned2025-12-15T17:47:16+00:00
URL https://security.org/robots.txt
Redirect https://www.security.org:443/robots.txt
Redirect Domain www.security.org
Redirect Base security.org
Domain IPs 3.165.75.38, 3.165.75.69, 3.165.75.7, 3.165.75.73
Redirect IPs 3.165.75.38, 3.165.75.69, 3.165.75.7, 3.165.75.73
Response IP 3.165.75.69
Found Yes
Hash 30b541f5b48ab0f5ce20f40a7849de1de274ccdadbc4d63f9acdce17b4768d5b
SimHash 7500d909e6a3

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /search/
Disallow /page/
Disallow /category/
Disallow /go/
Disallow /*?s=*
Disallow /*?action=trigger
Disallow /*?action=get_promotion
Disallow /*?s17
Disallow /wp-json
Disallow /wp-login.php
Disallow */hsimp/*.json

oai-searchbot
chatgpt-user
chatgpt-user/2.0
gptbot
anthropic-ai
claudebot
claude-web
perplexitybot
perplexity-user
google-extended
bingbot
amazonbot
applebot
applebot-extended
facebookbot
meta-externalagent
linkedinbot
duckassistbot

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.security.org/sitemap_index.xml

Comments

  • Global rules – apply to all crawlers
  • Explicit allow for AI/Chatbot-focused crawlers
  • Sitemap pointers