ski.com.au
robots.txt

Robots Exclusion Standard data for ski.com.au

Resource Scan

Scan Details

Site Domain ski.com.au
Base Domain ski.com.au
Scan Status Ok
Last Scan2024-11-13T20:21:59+00:00
Next Scan 2024-11-20T20:21:59+00:00

Last Scan

Scanned2024-11-13T20:21:59+00:00
URL https://ski.com.au/robots.txt
Redirect https://www.ski.com.au/robots.txt
Redirect Domain www.ski.com.au
Redirect Base ski.com.au
Domain IPs 13.237.67.13, 2406:da1c:2e1:d600:1d66:339d:5ce7:c998
Redirect IPs 13.237.67.13, 2406:da1c:2e1:d600:1d66:339d:5ce7:c998
Response IP 13.237.67.13
Found Yes
Hash d082c1af56aed0f42d42947663239b3af9113e8134cef8a854637671b9471e12
SimHash 1c35d3c1f580

Groups

bingbot

Rule Path
Allow /
Disallow /xf/attachments
Disallow /xf/goto
Disallow /xf/members
Disallow /xf/posts/*/reactions
Disallow /xf/uix

googlebot

Rule Path
Allow /
Disallow /xf/attachments
Disallow /xf/goto
Disallow /xf/members
Disallow /xf/posts/*/reactions
Disallow /xf/uix

adsbot-google

Rule Path
Allow /
Disallow /xf/attachments
Disallow /xf/goto
Disallow /xf/members
Disallow /xf/posts/*/reactions
Disallow /xf/uix

adsbot-google-mobile

Rule Path
Allow /
Disallow /xf/attachments
Disallow /xf/goto
Disallow /xf/members
Disallow /xf/posts/*/reactions
Disallow /xf/uix

mediapartners-google

Rule Path
Allow /
Disallow /xf/attachments
Disallow /xf/goto
Disallow /xf/members
Disallow /xf/posts/*/reactions
Disallow /xf/uix

applebot

Rule Path
Allow /
Disallow /xf/attachments
Disallow /xf/goto
Disallow /xf/members
Disallow /xf/posts/*/reactions
Disallow /xf/uix

ia_archiver

Rule Path
Allow /
Disallow /xf/attachments
Disallow /xf/goto
Disallow /xf/members
Disallow /xf/posts/*/reactions
Disallow /xf/uix

facebookexternalhit

Rule Path
Allow /*?*smid=

twitterbot

Rule Path
Allow /*?*smid=

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

buck

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googleother

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

openai

Rule Path
Disallow /

paqlebot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

scoop.it

Rule Path
Disallow /

seekr

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

verity

Rule Path
Disallow /

yandex

Rule Path
Disallow /

youbot

Rule Path
Disallow /

*

Rule Path
Disallow /

Comments

  • ALLOW
  • Bing
  • Google
  • Apple
  • Internet Archive
  • Social Link Retrieval
  • DISALLOW
  • Common Crawl
  • Moz
  • Google AI
  • Brandwatch
  • Majestic
  • Comscore
  • GumGum
  • All remaining obedient f*Bots