bexrealty.com
robots.txt
Robots Exclusion Standard data for bexrealty.com
Resource Scan
Scan Details
| Site Domain | bexrealty.com |
| Base Domain | bexrealty.com |
| Scan Status | Ok |
| Last Scan | 2024-11-14T10:20:25+00:00 |
| Next Scan | 2024-11-21T10:20:25+00:00 |
Last Scan
| Scanned | 2024-11-14T10:20:25+00:00 |
| URL | https://bexrealty.com/robots.txt |
| Redirect | https://www.bexrealty.com:443/robots.txt |
| Redirect Domain | www.bexrealty.com |
| Redirect Base | bexrealty.com |
| Domain IPs | 35.241.5.78 |
| Redirect IPs | 3.164.85.101, 3.164.85.29, 3.164.85.4, 3.164.85.98 |
| Response IP | 18.165.140.14 |
| Found | Yes |
| Hash | d0f3e9a04641776c97bb71aac9427cc8f95b0483198b58f22d1fe8eb169f1295 |
| SimHash | 721ca920ffe3 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | |
| Disallow | */map/* |
| Disallow | /support |
| Disallow | /documents.php* |
| Disallow | /fb3.php |
| Disallow | /logout.php |
| Disallow | /real_estate/*Privacy.php |
| Disallow | /real_estate/*privacy.php |
| Disallow | /*Privacy.php |
| Disallow | /*privacy.php |
| Disallow | /real_estate/*Terms.php |
| Disallow | /real_estate/*terms.php |
| Disallow | /*Terms.php |
| Disallow | /*terms.php |
| Disallow | /notices/* |
| Disallow | /*contact.php |
| Disallow | /*Condoreduced* |
| Disallow | /*Condoshortsale* |
| Disallow | /*Nreduced* |
| Disallow | /*Oceanfrontreduced* |
| Disallow | /*page_number%3D-* |
| Disallow | /*propertype%3DSingle%2BFamily* |
| Disallow | /*save%3D1* |
| Disallow | /*v4.att.php* |
| Disallow | /*v4.downloads.php* |
| Disallow | /*v4.email.php* |
| Disallow | /*v4.lead.php* |
| Disallow | /*v4.leaddetail.php* |
| Disallow | /*admin* |
| Disallow | /*unsubscribe.php* |
| Disallow | /*Yespropertype* |
| Disallow | /*Ypage_number* |
| Disallow | /*Ypropertype* |
| Disallow | /*Yreduced* |
| Disallow | /*Yshortsale* |
| Disallow | /*Ywaterfront* |
| Disallow | /*.htm |
| Disallow | /*.html |
| Disallow | /*%3Dy.php* |
| Disallow | /400.shtml |
| Disallow | /*eprice%3D* |
| Disallow | /*newlistings%3D* |
| Disallow | /*SI%3D* |
| Disallow | /*BUS_CENSUS_NAME* |
aboundex
add catalog
adnormcrawler
ahrefsbot
ahrefsbot.feeds
aihitbot
archive.org_bot
babya discoverer
botify
barkrowler
bbbot
blexbot
botw spider
bubing
butterfly/1.0
catchbot
ccbot
cigiema spider
compspybot
contextad bot 1.0
covenant eyes
crowsnest
css certificate spider
cyveillance
datanyze
dataprovider/6.92
daumoa 4.0
discobot
discoverybot
exabot
ezooms/1.0
feedburner
feedfetcher-google
findlinks
findxbot
fuzz faster u fool
flightdeckreportsbot
goodbot
geedobot
genieo/1.0
gigablastopensource
grapeshot
hubspot webcrawler
indy library
js-kit url resolver
ip-web-crawler
istellabot
konqueror
kraken
linkdexbot/2.0
lipperhey link explorer
meanpathbot
megaindex.com
megaindex.ru
merchantcentricbot
metauri
mj12bot
moreoverbot
ncbot
nerdybot
netcraftsurveyagent
netseer crawler
nominet.org
panscient.com
paperlibot
peerindex
percolatecrawler
proximic
queryseekerspider
ravencrawler
relateiq crawler
rogerbot
searchmetrics
scoutjet
scoutjet
scrapy/2.7.1
scrapy
screenerbot crawler
semalt crawler
semantifire1/0.20
semrushbot
seokicks-robot
seznambot
shodan.io
showyoubot
sistrix crawler
siteexplorer
skimbot/1.0
socialbm_bot
sogou web spider
solomonobot
sosospider
spbot
strawberryj.am
surveybot
tweetedtimes
tweetmemebot/3.0
twitterbot/1.0
umbot-ln/1.0
underscorebot
unwindfetchor
uptimerobot/2.0
urlappendbot
voltron
wbsearchbot
wotbox
xovibot
yahoo pipes 1.0
yrspider
zmeu
zscaler
petalbot
trovitbot
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.bexrealty.com/api/v1/sitemaps-new-listings-today/1.xml |
| sitemap | https://www.bexrealty.com/sitemaps-xml/master/1-new.xml.gz |
| sitemap | https://www.bexrealty.com/sitemaps-xml/master/1.xml.gz |
Warnings
- 2 invalid lines.