ratemyagent.co.nz
robots.txt

Robots Exclusion Standard data for ratemyagent.co.nz

Resource Scan

Scan Details

Site Domain ratemyagent.co.nz
Base Domain ratemyagent.co.nz
Scan Status Ok
Last Scan2024-11-14T07:59:30+00:00
Next Scan 2024-12-14T07:59:30+00:00

Last Scan

Scanned2024-11-14T07:59:30+00:00
URL https://ratemyagent.co.nz/robots.txt
Redirect https://www.ratemyagent.co.nz:443/robots.txt
Redirect Domain www.ratemyagent.co.nz
Redirect Base ratemyagent.co.nz
Domain IPs 13.225.4.110, 13.225.4.58, 13.225.4.75, 13.225.4.94
Redirect IPs 13.225.4.110, 13.225.4.58, 13.225.4.75, 13.225.4.94
Response IP 13.225.4.94
Found Yes
Hash a31f59589a240920d9763205796870bcce39516a1d8a3b9a6b2167a59f2cb8c7
SimHash 7b01d852ed00

Groups

*

Rule Path
Allow /
Disallow /error/
Disallow /404
Disallow /500
Disallow /dashboard/
Disallow /promoter/
Disallow /account
Disallow /agency/
Disallow /agent/
Disallow /suburb/search/
Disallow /testimonial/
Disallow /arkadia

applebot-extended
applebot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex
yandexbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

trendictionbot
trendiction

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot-sa
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ct
semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.ratemyagent.co.nz/sitemap.xml

Comments

  • https://babbar.tech/crawler
  • Majestic12