cleanhub.io
robots.txt

Robots Exclusion Standard data for cleanhub.io

Resource Scan

Scan Details

Site Domain cleanhub.io
Base Domain cleanhub.io
Scan Status Ok
Last Scan2025-09-09T07:42:58+00:00
Next Scan 2025-10-09T07:42:58+00:00

Last Scan

Scanned2025-09-09T07:42:58+00:00
URL https://cleanhub.io/robots.txt
Redirect https://www.cleanhub.com/robots.txt
Redirect Domain www.cleanhub.com
Redirect Base cleanhub.com
Domain IPs 104.26.14.186, 104.26.15.186, 172.67.70.76, 2606:4700:20::681a:eba, 2606:4700:20::681a:fba, 2606:4700:20::ac43:464c
Redirect IPs 104.26.6.201, 104.26.7.201, 172.67.70.71, 2606:4700:20::681a:6c9, 2606:4700:20::681a:7c9, 2606:4700:20::ac43:4647
Response IP 104.26.6.201
Found Yes
Hash 7244f125f7489ad75ae81db3e0d38ecf31a3299f46c79e2c29f9dcbb56a41a0f
SimHash 60146901e7f5

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /customer/
Disallow /embed/

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

applebot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

mistralai-user

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

Comments

  • — Bots you want to keep out