Log In Sign Up

productreview.com.au
robots.txt

Robots Exclusion Standard data for productreview.com.au

Resource Scan

Scan Details

Site Domain productreview.com.au
Base Domain productreview.com.au
Scan Status Ok
Last Scan2024-11-18T00:45:02+00:00
Next Scan 2024-11-25T00:45:02+00:00

Last Scan

Scanned2024-11-18T00:45:02+00:00
URL https://productreview.com.au/robots.txt
Redirect https://www.productreview.com.au/robots.txt
Redirect Domain www.productreview.com.au
Redirect Base productreview.com.au
Domain IPs 104.18.28.116, 104.18.29.116
Redirect IPs 104.18.28.116, 104.18.29.116
Response IP 104.18.29.116
Found Yes
Hash efe543df5a08b3a0487480b04e3412f08eedac4dcabc95dcb62879aea2449233
SimHash 70b9414980b0

Groups

discobot
exabot
ahrefsbot
mj12bot
netseer
proximic
psbot
trendictionbot
blexbot
sogou spider
mail.ru_bot/2.0
mail.ru
riddler
dotbot
superfeedr
sentibot
sentibot
bluechipbacklinks
applebot

Rule Path
Disallow /

amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
diffbot
facebookbot
friendlycrawler
gptbot
google-extended
googleother
googleother-image
googleother-video
icc-crawler
imagesiftbot
meta-externalagent
meta-externalfetcher
oai-searchbot
perplexitybot
petalbot
scrapy
timpibot
velenpublicwebcrawler
youbot
anthropic-ai
cohere-ai
facebookexternalhit
img2dataset
omgili
omgilibot

Rule Path
Disallow /

bingbot

Rule Path
Allow /listings/
Disallow /listings/*?
Allow /c/
Disallow /c/*?
Allow /b/
Disallow /b/*?
Allow /articles
Disallow /articles*?*
Disallow /listings/*/write-review
Disallow /

*

Rule Path
Disallow /api
Disallow /admin
Disallow /bm
Disallow /questions
Disallow /i/contact-us
Disallow /consumer-settings
Disallow /user-account-management
Disallow /write-review-confirm-email
Disallow /noop
Disallow /login?*
Disallow /4027029/
Back to top

Comments

  • https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
  • Google Ad / Analytics
Back to top