Log In Sign Up

bloghut.com.au
robots.txt

Robots Exclusion Standard data for bloghut.com.au

Resource Scan

Scan Details

Site Domain bloghut.com.au
Base Domain bloghut.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan3/2/2025, 12:52:12 AM
Next Scan 5/1/2025, 12:52:12 AM

Last Successful Scan

Scanned1/1/2025, 11:33:16 PM
URL https://bloghut.com.au/robots.txt
Domain IPs 104.21.89.118, 172.67.141.162, 2606:4700:3030::6815:5976, 2606:4700:3034::ac43:8da2
Response IP 104.21.89.118
Found Yes
Hash d4c7912b7914c25513609970f5b8add5f447a1dfb669fe59170bf6e87d5d54de
SimHash 741c5950c780

Groups

*

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /
Back to top