dubizzle.com.om
robots.txt

Robots Exclusion Standard data for dubizzle.com.om

Resource Scan

Scan Details

Site Domain dubizzle.com.om
Base Domain dubizzle.com.om
Scan Status Ok
Last Scan2024-11-04T10:25:30+00:00
Next Scan 2024-11-11T10:25:30+00:00

Last Scan

Scanned2024-11-04T10:25:30+00:00
URL https://dubizzle.com.om/robots.txt
Redirect https://www.dubizzle.com.om/robots.txt
Redirect Domain www.dubizzle.com.om
Redirect Base dubizzle.com.om
Domain IPs 108.156.133.129, 108.156.133.43, 108.156.133.47, 108.156.133.53
Redirect IPs 104.18.12.25, 104.18.13.25, 2606:4700::6812:c19, 2606:4700::6812:d19
Response IP 104.18.13.25
Found Yes
Hash 721a2f99c24cfa7f754644b3d22c1263e63d1be4137f4d03fa9cc8e30f7979be
SimHash 08110a604d13

Groups

*

Rule Path
Disallow /ajax/
Disallow /adminpanel/
Disallow /myaccount/
Disallow /adprint/
Disallow /ad/leaflet/
Disallow /ad/contact/
Disallow /payment/
Disallow /adding/confirm/
Disallow /adding/confirmpage/
Disallow /i2/ad/abuse/
Disallow /m/ad/abuse/
Disallow /apple-app-site-association
Disallow /profile/
Disallow /en/profile/
Disallow *ref%*
Disallow *search%*
Disallow *search%5B*
Disallow *sorting%3D*
Disallow *minPhotoCount%3D*
Disallow *product%3D*
Disallow */changelang/*
Disallow *method%3A*
Disallow *method%3A*
Disallow *?back=*
Disallow *himar%3D*
Disallow *_AM%3D%26*
Disallow *_PM%3D%26*
Disallow *?gdfp_req=*
Disallow *?invite=*
Disallow *?xtor=*
Disallow /account
Disallow /en/account
Disallow /captchaChallenge
Disallow /en/captchaChallenge
Allow /

*

Rule Path
Disallow /api/
Disallow /post/
Disallow /edit/
Disallow /chat/
Disallow /profile/
Disallow /payments/
Disallow /nf/
Disallow /deal_of_the_week/
Disallow /en/deal_of_the_week/
Disallow /featuredAgencies
Disallow /en/featuredAgencies
Disallow /api/
Disallow /recommended_properties*
Disallow /profolio/
Disallow /en/profolio/
Disallow /cp/
Disallow /en/cp/
Disallow /nova/
Disallow /en/nova/
Disallow *filter%3D*
Disallow */model-*/q-*
Disallow */raw-filters*

Comments

  • from olx.com.pk
  • explorer legacy
  • disallow all urls with parameters
  • disallow crawl on urls with model in path and free search
  • path on search urls used to retrieve filters as json