bayut.com
robots.txt
Robots Exclusion Standard data for bayut.com
Resource Scan
Scan Details
| Site Domain | bayut.com |
| Base Domain | bayut.com |
| Scan Status | Ok |
| Last Scan | 2024-09-28T17:06:33+00:00 |
| Next Scan | 2024-10-05T17:06:33+00:00 |
Last Scan
| Scanned | 2024-09-28T17:06:33+00:00 |
| URL | https://bayut.com/robots.txt |
| Redirect | https://www.bayut.com/robots.txt |
| Redirect Domain | www.bayut.com |
| Redirect Base | bayut.com |
| Domain IPs | 13.33.88.111, 13.33.88.124, 13.33.88.125, 13.33.88.69 |
| Redirect IPs | 54.230.71.105, 54.230.71.116, 54.230.71.120, 54.230.71.14 |
| Response IP | 13.225.4.79 |
| Found | Yes |
| Hash | 6ffcb77d4ed9761121adaefc9977acf74f328e0ba8a4ea2300abb7cf726f93b9 |
| SimHash | 443fade1c4e0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /stats/ |
| Disallow | /adserver/ |
| Disallow | /cp/ |
| Disallow | /search/ |
| Disallow | /nfpage/ |
| Disallow | /nf/ |
| Disallow | /trends/ |
| Disallow | /ar/trends/ |
| Disallow | /zh/trends/ |
| Disallow | /ru/trends/ |
| Disallow | /community/answers/ |
| Disallow | /community/discussions/ |
| Disallow | /bayut_v3/ |
| Disallow | /v3/ |
| Disallow | /international/ |
| Disallow | /bayut_0139b420/ |
| Disallow | /wanted/ |
| Disallow | /ar/wanted/ |
| Disallow | /zh/wanted/ |
| Disallow | /ru/wanted/ |
| Disallow | /file_uploader/ |
| Disallow | /feed_test_imp/ |
| Disallow | /blog/ |
| Disallow | /mybayut/unsubscribe/ |
| Disallow | /ajax/ |
| Disallow | /ar/ajax/ |
| Disallow | /zh/ajax/ |
| Disallow | /ru/ajax/ |
| Disallow | /deal_of_the_week/ |
| Disallow | /ar/deal_of_the_week/ |
| Disallow | /zh/deal_of_the_week/ |
| Disallow | /ru/deal_of_the_week/ |
| Disallow | /for-sale-ajax/ |
| Disallow | /to-rent-ajax/ |
| Disallow | /logout_ajax |
| Disallow | /ar/logout_ajax |
| Disallow | /zh/logout_ajax |
| Disallow | /ru/logout_ajax |
| Disallow | /getFloorPlans |
| Disallow | /ar/getFloorPlans |
| Disallow | /zh/getFloorPlans |
| Disallow | /ru/getFloorPlans |
| Disallow | /getAmenitiesList |
| Disallow | /ar/getAmenitiesList |
| Disallow | /zh/getAmenitiesList |
| Disallow | /ru/getAmenitiesList |
| Disallow | /getNearByLocations |
| Disallow | /ar/getNearByLocations |
| Disallow | /zh/getNearByLocations |
| Disallow | /ru/getNearByLocations |
| Disallow | /propertyList404/ |
| Disallow | /ar/propertyList404/ |
| Disallow | /zh/propertyList404/ |
| Disallow | /ru/propertyList404/ |
| Disallow | /getSimilarProperties |
| Disallow | /ar/getSimilarProperties |
| Disallow | /zh/getSimilarProperties |
| Disallow | /ru/getSimilarProperties |
| Disallow | /getAgentPropertiesList |
| Disallow | /ar/getAgentPropertiesList |
| Disallow | /zh/getAgentPropertiesList |
| Disallow | /ru/getAgentPropertiesList |
| Disallow | /ad-contact-btn |
| Disallow | /ad-mobile-app-btn |
| Disallow | /update_filters/ |
| Disallow | /featuredAgencies |
| Disallow | /ar/featuredAgencies |
| Disallow | /zh/featuredAgencies |
| Disallow | /ru/featuredAgencies |
| Disallow | /ar/mybayut/ |
| Disallow | /zh/mybayut/ |
| Disallow | /ru/mybayut/ |
| Disallow | /abu-dhabi/ |
| Disallow | /ajman/ |
| Disallow | /al-ain/ |
| Disallow | /dubai/ |
| Disallow | /fujairah/ |
| Disallow | /ras-al-khaimah/ |
| Disallow | /sharjah/ |
| Disallow | /umm-al-quwain/ |
| Disallow | /trend-location-list |
| Disallow | /ar/trend-location-list |
| Disallow | /zh/trend-location-list |
| Disallow | /ru/trend-location-list |
| Disallow | /videos_list |
| Disallow | /api/ |
| Disallow | /property/get_category_trends/ |
| Disallow | /property/get_popularity_trends/ |
| Disallow | /property/get_property_search_index/ |
| Disallow | /property/get_relative_agents/ |
| Disallow | /ar/property/get_category_trends/ |
| Disallow | /zh/property/get_category_trends/ |
| Disallow | /ru/property/get_category_trends/ |
| Disallow | /ar/property/get_popularity_trends/ |
| Disallow | /zh/property/get_popularity_trends/ |
| Disallow | /ru/property/get_popularity_trends/ |
| Disallow | /ar/property/get_property_search_index/ |
| Disallow | /zh/property/get_property_search_index/ |
| Disallow | /ru/property/get_property_search_index/ |
| Disallow | /ar/property/get_relative_agents/ |
| Disallow | /zh/property/get_relative_agents/ |
| Disallow | /ru/property/get_relative_agents/ |
| Disallow | /recommended_properties* |
| Disallow | /profolio/ |
| Disallow | /academy/ |
| Disallow | /account |
| Disallow | /ar/account |
| Disallow | /zh/account |
| Disallow | /ru/account |
| Disallow | /captchaChallenge |
| Disallow | /ar/captchaChallenge |
| Disallow | /zh/captchaChallenge |
| Disallow | /ru/captchaChallenge |
| Disallow | /registration.html |
| Disallow | /ar/registration.html |
| Disallow | /zh/registration.html |
| Disallow | /ru/registration.html |
| Allow | /api/areaGuideLink$ |
| Allow | /api/internalLinks/propertyMarketAnalysisPage/ |
| Allow | /api/truBroker/search |
facebookexternalhit/1.1
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 15 |
Warnings
- 2 invalid lines.