Log In Sign Up

sooperarticles.com
robots.txt

Robots Exclusion Standard data for sooperarticles.com

Resource Scan

Scan Details

Site Domain sooperarticles.com
Base Domain sooperarticles.com
Scan Status Ok
Last Scan2026-02-03T10:20:46+00:00
Next Scan 2026-02-10T10:20:46+00:00

Last Scan

Scanned2026-02-03T10:20:46+00:00
URL https://sooperarticles.com/robots.txt
Domain IPs 104.26.0.205, 104.26.1.205, 172.67.74.91, 2606:4700:20::681a:1cd, 2606:4700:20::681a:cd, 2606:4700:20::ac43:4a5b
Response IP 104.26.0.205
Found Yes
Hash 55bd521da60ef6475d5d5e4729f899b5ea73338d2026a6ebacdb68acb4b9a08f
SimHash 5830ad10e4ba

Groups

gptbot

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

chatgpt-user

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

ccbot

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

anthropic-ai

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

claude-web

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

bytespider

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

perplexitybot

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

google-extended

Rule Path
Disallow /search
Disallow /search/
Disallow /tag/
Disallow /author/

*

Rule Path
Disallow /api/
Back to top

Other Records

Field Value
sitemap https://www.sooperarticles.com/sitemap.xml
Back to top

Comments

  • AI crawlers - block expensive pages only
  • Rules for all other bots
  • Sitemap
Back to top