Log In Sign Up

loanstreet.com.my
robots.txt

Robots Exclusion Standard data for loanstreet.com.my

Resource Scan

Scan Details

Site Domain loanstreet.com.my
Base Domain loanstreet.com.my
Scan Status Ok
Last Scan2024-09-30T02:52:14+00:00
Next Scan 2024-10-07T02:52:14+00:00

Last Scan

Scanned2024-09-30T02:52:14+00:00
URL https://loanstreet.com.my/robots.txt
Domain IPs 139.162.50.39
Response IP 139.162.50.39
Found Yes
Hash b79754ce9a1f29cfde45ded98c956b697124fa1a0dd6096af6aa137bc5d88c89
SimHash 208c5c7ca5e0

Groups

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /backend/
Disallow /plugins/
Disallow /articles/
Disallow /admin/
Disallow /includes/
Disallow /learning-centre/page/
Disallow /learning-centre/search
Disallow /tags/
Disallow /ms/tags/
Disallow /widget/
Disallow /ms/pusat-pembelajaran/laman/
Disallow /doc/
Disallow /simple_captcha
Disallow /subscription/
Disallow /ms/subscription/
Disallow /insurance/motorcycle-insurance-road-tax/compare-price
Disallow /insurance/car-insurance-road-tax/compare-price
Disallow /insurance/ms/insurans-roadtax-kereta/banding-harga
Disallow /insurance/ms/insurans-roadtax-motosikal/banding-harga
Disallow /index.htm
Disallow /ms/kad-kr%E2%80%8Bedit/aeon-bank
Disallow /personal-%E2%80%8Bloans
Disallow /credit-card/*/apply
Disallow /ms/kad-kredit/*/mohon
Disallow */verify_otp
Disallow /insurance/*/sebut-harga
Disallow /insurance/*/quotation
Disallow /home-loan/verify_otp
Disallow /documents/
Allow /home-loan/*
Disallow /*?
Back to top

Other Records

Field Value
sitemap https://www.loanstreet.com.my/en/sitemaps/sitemap.xml.gz
sitemap https://www.loanstreet.com.my/ms/sitemaps/sitemap.xml.gz
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-Agent: *
  • Disallow: /
  • allows mortgage urls with query string to be indexed by crawlers
Back to top

Warnings

  • 4 invalid lines.
Back to top