cleanhub.com
robots.txt

Robots Exclusion Standard data for cleanhub.com

Resource Scan

Scan Details

Site Domain cleanhub.com
Base Domain cleanhub.com
Scan Status Ok
Last Scan2025-09-26T11:13:14+00:00
Next Scan 2025-10-26T11:13:14+00:00

Last Scan

Scanned2025-09-26T11:13:14+00:00
URL https://cleanhub.com/robots.txt
Redirect https://www.cleanhub.com/robots.txt
Redirect Domain www.cleanhub.com
Redirect Base cleanhub.com
Domain IPs 104.26.6.201, 104.26.7.201, 172.67.70.71, 2606:4700:20::681a:6c9, 2606:4700:20::681a:7c9, 2606:4700:20::ac43:4647
Redirect IPs 104.26.6.201, 104.26.7.201, 172.67.70.71, 2606:4700:20::681a:6c9, 2606:4700:20::681a:7c9, 2606:4700:20::ac43:4647
Response IP 104.26.6.201
Found Yes
Hash 7244f125f7489ad75ae81db3e0d38ecf31a3299f46c79e2c29f9dcbb56a41a0f
SimHash 60146901e7f5

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /customer/
Disallow /embed/

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

applebot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

mistralai-user

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

Comments

  • — Bots you want to keep out