Log In Sign Up

penzu.com.au
robots.txt

Robots Exclusion Standard data for penzu.com.au

Resource Scan

Scan Details

Site Domain penzu.com.au
Base Domain penzu.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan4/26/2025, 11:44:51 AM
Next Scan 7/25/2025, 11:44:51 AM

Last Successful Scan

Scanned12/27/2024, 12:33:32 PM
URL https://penzu.com.au/robots.txt
Domain IPs 104.21.78.183, 172.67.168.84, 2606:4700:3032::ac43:a854, 2606:4700:3034::6815:4eb7
Response IP 104.21.78.183
Found Yes
Hash d4c7912b7914c25513609970f5b8add5f447a1dfb669fe59170bf6e87d5d54de
SimHash 741c5950c780

Groups

*

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /
Back to top