ratemyagent.co.nz
robots.txt

Robots Exclusion Standard data for ratemyagent.co.nz

Resource Scan

Scan Details

Site Domain ratemyagent.co.nz
Base Domain ratemyagent.co.nz
Scan Status Ok
Last Scan2024-05-18T07:57:09+00:00
Next Scan 2024-06-17T07:57:09+00:00

Last Scan

Scanned2024-05-18T07:57:09+00:00
URL https://ratemyagent.co.nz/robots.txt
Redirect https://www.ratemyagent.co.nz:443/robots.txt
Redirect Domain www.ratemyagent.co.nz
Redirect Base ratemyagent.co.nz
Domain IPs 13.225.4.110, 13.225.4.58, 13.225.4.75, 13.225.4.94
Redirect IPs 13.225.4.110, 13.225.4.58, 13.225.4.75, 13.225.4.94
Response IP 13.225.4.75
Found Yes
Hash f26063f3827bd55d63ed58d2811ff9cb03186b9f7a105f51cf785b1d8499326b
SimHash 6b01d852ed10

Groups

*

Rule Path
Allow /
Disallow /error/
Disallow /404
Disallow /500
Disallow /dashboard/
Disallow /promoter/
Disallow /account
Disallow /agency/
Disallow /agent/
Disallow /suburb/search/
Disallow /testimonial/
Disallow /arkadia

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

semrushbot-sa
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ct
semrushbot-bm

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.ratemyagent.co.nz/sitemap.xml

Comments

  • https://babbar.tech/crawler