Log In Sign Up

blogstar.com.au
robots.txt

Robots Exclusion Standard data for blogstar.com.au

Resource Scan

Scan Details

Site Domain blogstar.com.au
Base Domain blogstar.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan3/4/2025, 7:38:07 AM
Next Scan 5/3/2025, 7:38:07 AM

Last Successful Scan

Scanned1/4/2025, 7:10:45 AM
URL https://blogstar.com.au/robots.txt
Domain IPs 104.21.80.23, 172.67.173.85, 2606:4700:3034::6815:5017, 2606:4700:3037::ac43:ad55
Response IP 104.21.80.23
Found Yes
Hash d4c7912b7914c25513609970f5b8add5f447a1dfb669fe59170bf6e87d5d54de
SimHash 741c5950c780

Groups

*

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /
Back to top