olx.com.kw
robots.txt

Robots Exclusion Standard data for olx.com.kw

Resource Scan

Scan Details

Site Domain olx.com.kw
Base Domain olx.com.kw
Scan Status Ok
Last Scan2024-06-12T05:02:06+00:00
Next Scan 2024-06-19T05:02:06+00:00

Last Scan

Scanned2024-06-12T05:02:06+00:00
URL https://olx.com.kw/robots.txt
Redirect https://www.dubizzle.com.kw/robots.txt
Redirect Domain www.dubizzle.com.kw
Redirect Base dubizzle.com.kw
Domain IPs 54.230.71.124, 54.230.71.41, 54.230.71.5, 54.230.71.96
Redirect IPs 104.18.33.104, 172.64.154.152, 2606:4700:4400::6812:2168, 2606:4700:4400::ac40:9a98
Response IP 172.64.154.152
Found Yes
Hash 83d4760a30f599834b4a717b80e6015cfdfc0f12dd884218dd0cfa733aaebd59
SimHash 289108600d32

Groups

*

Rule Path
Disallow /ajax/
Disallow /adminpanel/
Disallow /myaccount/
Disallow /adprint/
Disallow /ad/leaflet/
Disallow /ad/contact/
Disallow /payment/
Disallow /adding/confirm/
Disallow /adding/confirmpage/
Disallow /i2/ad/abuse/
Disallow /m/ad/abuse/
Disallow /apple-app-site-association
Disallow /profile/
Disallow /en/profile/
Disallow *ref%*
Disallow *search%*
Disallow *search%5B*
Disallow *sorting%3D*
Disallow *minPhotoCount%3D*
Disallow *product%3D*
Disallow */changelang/*
Disallow /account
Disallow /en/account
Disallow /captchaChallenge
Disallow /en/captchaChallenge
Allow /

*

Rule Path
Disallow /api/
Disallow /post/
Disallow /edit/
Disallow /chat/
Disallow /profile/
Disallow /payments/
Disallow /nf/
Disallow /deal_of_the_week/
Disallow /en/deal_of_the_week/
Disallow /featuredAgencies
Disallow /en/featuredAgencies
Disallow /api/
Disallow /recommended_properties*
Disallow /profolio/
Disallow /en/profolio/
Disallow /cp/
Disallow /en/cp/
Disallow /nova/
Disallow /en/nova/
Disallow *filter%3D*
Disallow */model-*/q-*

Comments

  • from olx.com.pk
  • explorer legacy
  • disallow all urls with parameters
  • disallow crawl on urls with model in path and free search