dubizzle.qa
robots.txt

Robots Exclusion Standard data for dubizzle.qa

Resource Scan

Scan Details

Site Domain dubizzle.qa
Base Domain dubizzle.qa
Scan Status Ok
Last Scan2024-06-12T17:12:27+00:00
Next Scan 2024-06-19T17:12:27+00:00

Last Scan

Scanned2024-06-12T17:12:27+00:00
URL https://dubizzle.qa/robots.txt
Redirect https://www.dubizzle.qa/robots.txt
Redirect Domain www.dubizzle.qa
Redirect Base dubizzle.qa
Domain IPs 65.8.11.104, 65.8.11.106, 65.8.11.69, 65.8.11.83
Redirect IPs 104.18.36.153, 172.64.151.103, 2606:4700:4400::6812:2499, 2606:4700:4400::ac40:9767
Response IP 104.18.36.153
Found Yes
Hash c3278070fd6db8c41f6e1dd529b91871f465139bc2a435bf04a0bc9db2645ef6
SimHash 399108600d32

Groups

*

Rule Path
Disallow /ajax/
Disallow /adminpanel/
Disallow /myaccount/
Disallow /adprint/
Disallow /ad/leaflet/
Disallow /ad/contact/
Disallow /payment/
Disallow /adding/confirm/
Disallow /adding/confirmpage/
Disallow /i2/ad/abuse/
Disallow /m/ad/abuse/
Disallow /apple-app-site-association
Disallow /profile/
Disallow /en/profile/
Disallow *ref%*
Disallow *search%*
Disallow *search%5B*
Disallow *sorting%3D*
Disallow *minPhotoCount%3D*
Disallow *product%3D*
Disallow */changelang/*
Disallow *sort_by%3D*
Disallow *back%3D*
Disallow /account
Disallow /en/account
Disallow /captchaChallenge
Disallow /en/captchaChallenge
Allow /

*

Rule Path
Disallow /api/
Disallow /post/
Disallow /edit/
Disallow /chat/
Disallow /profile/
Disallow /payments/
Disallow /nf/
Disallow /deal_of_the_week/
Disallow /en/deal_of_the_week/
Disallow /featuredAgencies
Disallow /en/featuredAgencies
Disallow /api/
Disallow /recommended_properties*
Disallow /profolio/
Disallow /en/profolio/
Disallow /cp/
Disallow /en/cp/
Disallow /nova/
Disallow /en/nova/
Disallow *filter%3D*
Disallow */model-*/q-*

Comments

  • from olx.com.pk
  • explorer legacy
  • disallow all urls with parameters
  • disallow crawl on urls with model in path and free search