placebuzz.com
robots.txt

Robots Exclusion Standard data for placebuzz.com

Resource Scan

Scan Details

Site Domain placebuzz.com
Base Domain placebuzz.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-31T07:55:14+00:00
Next Scan 2024-11-14T07:55:14+00:00

Last Successful Scan

Scanned2024-10-16T07:54:14+00:00
URL https://placebuzz.com/robots.txt
Redirect https://www.placebuzz.com/robots.txt
Redirect Domain www.placebuzz.com
Redirect Base placebuzz.com
Domain IPs 13.33.30.13, 13.33.30.25, 13.33.30.33, 13.33.30.37
Redirect IPs 54.230.112.108, 54.230.112.120, 54.230.112.121, 54.230.112.124
Response IP 3.164.68.41
Found Yes
Hash bd79b6a7ec40d890742bf4eac313fe2b75af16a07920ae14efd1efc9b2ca94ae
SimHash c1f70b10e179

Groups

*

Rule Path
Disallow /*SearchArea%3D
Disallow /*utm_source%3D
Disallow /*PlaceSearchCriteriaId%3D
Disallow /*Radius%3D
Disallow /*MinPrice%3D
Disallow /*MaxPrice%3D
Disallow /*SortType%3D
Disallow /*?MinNumberOfBedroomsId=
Disallow /*?MaxNumberOfBedroomsId=
Disallow /property-for-sale/details/*/images
Disallow /property-for-rent/details/*/images
Disallow /property-for-sale/details/*/images/*
Disallow /property-for-rent/details/*/images/*
Disallow /property-for-sale/details/*/contact-agent
Disallow /property-for-rent/details/*/contact-agent
Disallow /property-for-sale/details/*/documents
Disallow /property-for-rent/details/*/documents
Disallow /property-for-sale/details/*/floorplans
Disallow /property-for-rent/details/*/floorplans
Disallow /photo/

baiduspider

Rule Path
Disallow /

bingbot

Rule Path
Disallow /property-for-rent/details/
Disallow /property-for-sale/details/

Other Records

Field Value
crawl-delay 5

admantx

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

proximic

Rule Path
Disallow /

slurp

Rule Path
Disallow /

grapeshot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

petalbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

spriftcrawler

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

cbbot

Rule Path
Disallow /

oodlebot

Rule Path
Disallow /

spider

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

verity/1.1

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

Comments

  • Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
  • blocked as they are making incorrect requests
  • Slurp
  • let grapeshot crawl all pages
  • Petalbot blocked as they are making a huge number of requests