Log In Sign Up

priceme.com.au
robots.txt

Robots Exclusion Standard data for priceme.com.au

Resource Scan

Scan Details

Site Domain priceme.com.au
Base Domain priceme.com.au
Scan Status Ok
Last Scan2024-11-11T19:56:25+00:00
Next Scan 2024-11-18T19:56:25+00:00

Last Scan

Scanned2024-11-11T19:56:25+00:00
URL https://priceme.com.au/robots.txt
Redirect https://www.priceme.com.au/robots.txt
Redirect Domain www.priceme.com.au
Redirect Base priceme.com.au
Domain IPs 104.26.12.76, 104.26.13.76, 172.67.74.197, 2606:4700:20::681a:c4c, 2606:4700:20::681a:d4c, 2606:4700:20::ac43:4ac5
Redirect IPs 104.26.12.76, 104.26.13.76, 172.67.74.197, 2606:4700:20::681a:c4c, 2606:4700:20::681a:d4c, 2606:4700:20::ac43:4ac5
Response IP 172.67.74.197
Found Yes
Hash 7f850f2cda9fb1899bf10fa6d25d55bda2175e42cdfdaf9578b1ecc522bc4279
SimHash e01c509ace95

Groups

*

Rule Path
Disallow /*/*%7D%7D$
Disallow /*.axd$
Disallow /shops-nearby
Disallow /mycart*
Disallow /privacypolicy
Disallow /Feedback?*
Disallow /blog/wp-login.php*
Disallow /RetailerReview/
Disallow /.axd$
Disallow /JudgementLogin
Disallow /JudgementLogin*
Disallow /GetCatalogfav
Disallow /GetProductfav
Disallow /GetProductList
Disallow /consumer/feed
Disallow /consumer/wp-admin/
Disallow /money/termdepositspostback.aspx*
Disallow /money/savingsaccountspostback.aspx*
Disallow /Compare.aspx*

blexbot

Rule Path
Disallow /

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seznambot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

teemer

Rule Path
Disallow /

netseer

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

eurobot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

wget

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

onet.pl

Rule Path
Disallow /

yandex

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

cyberpatrol

Rule Path
Disallow /

houxoucrawler

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

almaden

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

dumbbot

Rule Path
Disallow /

generic

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

nexabot

Rule Path
Disallow /

npbot

Rule Path
Disallow /

owr_crawler

Rule Path
Disallow /

rabaz

Rule Path
Disallow /

rpt-httpclient

Rule Path
Disallow /

scoutabout

Rule Path
Disallow /

semanticdiscovery

Rule Path
Disallow /

cherry picker

Rule Path
Disallow /

crescent internet toolpak

Rule Path
Disallow /

disco pump

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

teleport

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

ultraseek

Rule Path
Disallow /

vspider

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webemailextractor

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

israbot

Rule Path
Disallow

orthogaffe

Rule Path
Disallow

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

grub

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

htdig

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

netprospector

Rule Path
Disallow /

perman surfer

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

yahooseeker

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

e-societyrobot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.priceme.com.au/sitemap/SiteMap.xml
Back to top

Comments

  • file for https://www.priceme.com.au
  • disallow rules for AI crawlers (ChatGPT, Claude etc.)
Back to top