Log In Sign Up

aussieanimals.com
robots.txt

Robots Exclusion Standard data for aussieanimals.com

Resource Scan

Scan Details

Site Domain aussieanimals.com
Base Domain aussieanimals.com
Scan Status Ok
Last Scan2026-02-23T04:18:12+00:00
Next Scan 2026-03-02T04:18:12+00:00

Last Scan

Scanned2026-02-23T04:18:12+00:00
URL https://aussieanimals.com/robots.txt
Domain IPs 104.26.10.210, 104.26.11.210, 172.67.69.8, 2606:4700:20::681a:ad2, 2606:4700:20::681a:bd2, 2606:4700:20::ac43:4508
Response IP 104.26.11.210
Found Yes
Hash 7774dbd5c49614771f5f0cbcfd67ef8dca3c520db92f864529d7a69792c42b34
SimHash 70186bc4d5dd

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

google-inspectiontool

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

applebot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

facebot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexitycrawler

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

youbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

gptsitecrawler

Rule Path
Disallow /

writer

Rule Path
Disallow /

quora-link-preview

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://aussieanimals.com/sitemaps.xml
Back to top

Comments

  • ===============================
  • robots.txt for AussieAnimals.com
  • Purpose: Allow search/ads bots; block AI training crawlers
  • ===============================
  • --- Sitemaps & monetisation ---
  • --- Explicit ALLOW (search + ads + tools) ---
  • --- Block popular AI/LLM crawlers (training/scraping) ---
  • --- Default fallback ---
  • Notes:
  • - Keeping * default Allow to avoid collateral SEO damage.
  • - LLM/AI lists evolve; add new UAs you see in logs.
  • - Some scrapers ignore robots.txt—use WAF/rate limits to enforce.
Back to top

Warnings

  • 1 invalid line.
Back to top