propertynews.com
robots.txt

Robots Exclusion Standard data for propertynews.com

Resource Scan

Scan Details

Site Domain propertynews.com
Base Domain propertynews.com
Scan Status Ok
Last Scan2024-11-15T09:15:38+00:00
Next Scan 2024-11-22T09:15:38+00:00

Last Scan

Scanned2024-11-15T09:15:38+00:00
URL https://propertynews.com/robots.txt
Domain IPs 2600:9000:20fe:1200:3:ee7a:15c0:93a1, 2600:9000:20fe:1400:3:ee7a:15c0:93a1, 2600:9000:20fe:2000:3:ee7a:15c0:93a1, 2600:9000:20fe:2e00:3:ee7a:15c0:93a1, 2600:9000:20fe:7600:3:ee7a:15c0:93a1, 2600:9000:20fe:7c00:3:ee7a:15c0:93a1, 2600:9000:20fe:8c00:3:ee7a:15c0:93a1, 2600:9000:20fe:f200:3:ee7a:15c0:93a1, 52.222.144.107, 52.222.144.12, 52.222.144.36, 52.222.144.87
Response IP 65.9.112.43
Found Yes
Hash 5b20a4707dc3039785072a12c04bb31273b600af3a94e67843188dbbecc52250
SimHash 78fc9a002993

Groups

*

Rule Path
Disallow *?status%5B%5D*
Disallow *?relevance=*
Disallow *?sort=*
Disallow *Keywords%3D*
Disallow /8216/www.propertynews/*
Disallow /places
Disallow /favourites
Disallow /searches
Disallow /hidden
Disallow /notes
Disallow /profile
Disallow /login
Disallow /results/*
Disallow /members/*

trovitbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

wesee

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandex bot

Rule Path
Disallow /

gurujibot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou pic spider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

oozbot

Rule Path
Disallow /

tagoobot

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

saladspoon

Rule Path
Disallow /

temaseek.com

Rule Path
Disallow /

ahrefbot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.propertynews.com/static/sitemap_index_high_freq.xml
sitemap https://www.propertynews.com/static/sitemap_index_low_freq.xml

Comments

  • Internal Search Function & Logged in Users
  • Unwanted Crawlers
  • Sitemap Index

Warnings

  • 2 invalid lines.