propertynews.com
robots.txt

Robots Exclusion Standard data for propertynews.com

Resource Scan

Scan Details

Site Domain propertynews.com
Base Domain propertynews.com
Scan Status Ok
Last Scan2024-06-14T06:30:52+00:00
Next Scan 2024-06-21T06:30:52+00:00

Last Scan

Scanned2024-06-14T06:30:52+00:00
URL https://propertynews.com/robots.txt
Domain IPs 2600:9000:20a6:7800:3:ee7a:15c0:93a1, 2600:9000:20a6:9000:3:ee7a:15c0:93a1, 2600:9000:20a6:9a00:3:ee7a:15c0:93a1, 2600:9000:20a6:a000:3:ee7a:15c0:93a1, 2600:9000:20a6:b400:3:ee7a:15c0:93a1, 2600:9000:20a6:d200:3:ee7a:15c0:93a1, 2600:9000:20a6:dc00:3:ee7a:15c0:93a1, 2600:9000:20a6:e000:3:ee7a:15c0:93a1, 99.84.203.111, 99.84.203.7, 99.84.203.80, 99.84.203.92
Response IP 3.160.246.60
Found Yes
Hash 5b20a4707dc3039785072a12c04bb31273b600af3a94e67843188dbbecc52250
SimHash 78fc9a002993

Groups

*

Rule Path
Disallow *?status%5B%5D*
Disallow *?relevance=*
Disallow *?sort=*
Disallow *Keywords%3D*
Disallow /8216/www.propertynews/*
Disallow /places
Disallow /favourites
Disallow /searches
Disallow /hidden
Disallow /notes
Disallow /profile
Disallow /login
Disallow /results/*
Disallow /members/*

trovitbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

wesee

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandex bot

Rule Path
Disallow /

gurujibot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou pic spider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

oozbot

Rule Path
Disallow /

tagoobot

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

saladspoon

Rule Path
Disallow /

temaseek.com

Rule Path
Disallow /

ahrefbot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.propertynews.com/static/sitemap_index_high_freq.xml
sitemap https://www.propertynews.com/static/sitemap_index_low_freq.xml

Comments

  • Internal Search Function & Logged in Users
  • Unwanted Crawlers
  • Sitemap Index

Warnings

  • 2 invalid lines.