olx.sa.com
robots.txt

Robots Exclusion Standard data for olx.sa.com

Resource Scan

Scan Details

Site Domain olx.sa.com
Base Domain olx.sa.com
Scan Status Ok
Last Scan2024-11-16T13:50:15+00:00
Next Scan 2024-11-23T13:50:15+00:00

Last Scan

Scanned2024-11-16T13:50:15+00:00
URL https://olx.sa.com/robots.txt
Redirect https://www.dubizzle.sa/robots.txt
Redirect Domain www.dubizzle.sa
Redirect Base dubizzle.sa
Domain IPs 13.33.88.30, 13.33.88.8, 13.33.88.88, 13.33.88.95
Redirect IPs 104.18.36.246, 172.64.151.10, 2606:4700:4400::6812:24f6, 2606:4700:4400::ac40:970a
Response IP 172.64.151.10
Found Yes
Hash c7fb63aac27b3125a0b074914865d1f0a17e2b7b00027350bc3e46f188b6843a
SimHash 28170f614d13

Groups

*

Rule Path
Disallow /ajax/
Disallow /adminpanel/
Disallow /myaccount/
Disallow /adprint/
Disallow /ad/leaflet/
Disallow /ad/contact/
Disallow /payment/
Disallow /adding/confirm/
Disallow /adding/confirmpage/
Disallow /i2/ad/abuse/
Disallow /m/ad/abuse/
Disallow /apple-app-site-association
Disallow /profile/
Disallow /en/profile/
Disallow *ref%*
Disallow *search%*
Disallow *search%5B*
Disallow *sorting%3D*
Disallow *minPhotoCount%3D*
Disallow *product%3D*
Disallow */changelang/*
Disallow /account
Disallow /ar/account
Disallow /captchaChallenge
Disallow /ar/captchaChallenge
Allow /

*

Rule Path
Disallow /api/
Disallow /post/
Disallow /edit/
Disallow /chat/
Disallow /profile/
Disallow /payments/
Disallow /nf/
Disallow /deal_of_the_week/
Disallow /en/deal_of_the_week/
Disallow /featuredAgencies
Disallow /en/featuredAgencies
Disallow /api/
Disallow /recommended_properties*
Disallow /profolio/
Disallow /en/profolio/
Disallow /cp/
Disallow /en/cp/
Disallow /nova/
Disallow /en/nova/
Disallow *filter%3D*
Disallow */model-*/q-*
Disallow */raw-filters*

Comments

  • from olx.com.pk
  • explorer legacy
  • disallow all urls with parameters
  • disallow crawl on urls with model in path and free search
  • path on search urls used to retrieve filters as json