Log In Sign Up

smartcompany.com.au
robots.txt

Robots Exclusion Standard data for smartcompany.com.au

Resource Scan

Scan Details

Site Domain smartcompany.com.au
Base Domain smartcompany.com.au
Scan Status Ok
Last Scan2024-09-25T18:22:58+00:00
Next Scan 2024-10-02T18:22:58+00:00

Last Scan

Scanned2024-09-25T18:22:58+00:00
URL https://smartcompany.com.au/robots.txt
Redirect https://www.smartcompany.com.au/robots.txt
Redirect Domain www.smartcompany.com.au
Redirect Base smartcompany.com.au
Domain IPs 192.0.66.219, 2a04:fa87:fffd::c000:42db
Redirect IPs 13.224.163.14, 13.224.163.24, 13.224.163.31, 13.224.163.33
Response IP 3.165.82.39
Found Yes
Hash 89ef3f65890c64058710fb6be6d48e0bb680c566285d982305f0dd53760501b9
SimHash 7118c8c0a412

Groups

*

Rule Path
Disallow /wp-admin/

ia_archiver

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

cohere-ai

Rule Path Comment
Disallow / START YOAST BLOCK

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.smartcompany.com.au/sitemap_index.xml
Back to top

Comments

  • ---------------------------
  • ---------------------------
  • Block CommonCrawl
  • ---------------------------
  • Block OpenAI
  • ---------------------------
  • Block Bard / Gemini crawler
  • ---------------------------
  • Block Perplexity AI
  • ---------------------------
  • Block Anthropic AI
  • ---------------------------
  • Block Webz.io
  • ---------------------------
  • Block Meta
  • ---------------------------
  • Block Apple
  • ---------------------------
  • Block ByteDance
  • ---------------------------
  • Block Diffbot
  • ---------------------------
  • Block Imagesift by Hive
  • ---------------------------
  • Block YouBot
  • ---------------------------
  • Block cohere
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
Back to top