idp.com
robots.txt

Robots Exclusion Standard data for idp.com

Resource Scan

Scan Details

Site Domain idp.com
Base Domain idp.com
Scan Status Ok
Last Scan2024-10-20T07:11:43+00:00
Next Scan 2024-11-19T07:11:43+00:00

Last Scan

Scanned2024-10-20T07:11:43+00:00
URL https://idp.com/robots.txt
Redirect https://www.idp.com/robots.txt
Redirect Domain www.idp.com
Redirect Base idp.com
Domain IPs 3.214.249.60, 34.199.65.120, 54.209.40.233
Redirect IPs 108.157.254.37, 108.157.254.45, 108.157.254.66, 108.157.254.80
Response IP 108.157.254.66
Found Yes
Hash 83145f2084b6562b329a2275e2971ede2784d98273cdb2f5c8f0a9a141f1f4d4
SimHash 0f2452020f78

Groups

*

Rule Path
Disallow /*topics%3D
Disallow /*session_logout%3D
Disallow /*page%3D1$
Disallow /*destination_country%3D
Disallow /*stage%3D
Disallow /*fit%3D
Disallow /*sort%3D
Disallow /student-essentials/*lang%3Den
Disallow /australia/*lang%3Den
Disallow /cambodia/*lang%3Den
Disallow /canada/*lang%3Den
Disallow /egypt/*lang%3Den
Disallow /hongkong/*lang%3Dzk
Disallow /indonesia/*lang%3Did
Disallow /iran/*lang%3Dfa
Disallow /middleeast/*lang%3Dar
Disallow /korea/*lang%3Dko
Disallow /oman/*lang%3Den
Disallow /saudiarabia/*lang%3Dar
Disallow /singapore/*lang%3Den
Disallow /taiwan/*lang%3Dtw
Disallow /thailand/*lang%3Dth
Disallow /turkey/*lang%3Dtr
Disallow /uae/*lang%3Den
Disallow /vietnam/*lang%3Dvn
Disallow /*/cart
Disallow /*/checkout
Disallow /*/my-account
Disallow /india/SearchBox/
Disallow isChreq
Disallow /india/NA$
Disallow %26%26
Disallow country%3D
Disallow dt%3D
Disallow destinationcountries%3D
Disallow item%3D
Disallow instituteId%3D
Disallow institutionId%3D
Disallow /*/~/
Disallow */search/courses/*?*
Disallow */search*?*q*
Disallow */search?query=
Disallow /*/campaigns
Allow /australia/events/adelaide/$
Disallow /australia/events/adelaide/*
Allow /australia/events/brisbane/$
Disallow /australia/events/brisbane/*
Allow /australia/events/melbourne/$
Disallow /australia/events/melbourne/*
Allow /australia/events/perth/$
Disallow /australia/events/perth/*
Allow /australia/events/sydney/$
Disallow /australia/events/sydney/*
Allow /australia/events/parramatta/$
Disallow /australia/events/parramatta/*

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

ahrefssiteaudit

Rule Path
Disallow /

addthis.com robot

Rule Path
Disallow /

botw spider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

commoncrawler

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

yahoo pipes 2.0

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.idp.com/sitemap.xml
sitemap https://www.idp.com/australia/sitemap.xml
sitemap https://www.idp.com/bangladesh/sitemap.xml
sitemap https://www.idp.com/cambodia/sitemap.xml
sitemap https://www.idp.com/canada/sitemap.xml
sitemap https://www.idp.com/egypt/sitemap.xml
sitemap https://www.idp.com/ghana/sitemap.xml
sitemap https://www.idp.com/hongkong/sitemap.xml
sitemap https://www.idp.com/india/sitemap.xml
sitemap https://www.idp.com/indonesia/sitemap.xml
sitemap https://www.idp.com/iran/sitemap.xml
sitemap https://www.idp.com/kenya/sitemap.xml
sitemap https://www.idp.com/korea/sitemap.xml
sitemap https://www.idp.com/malaysia/sitemap.xml
sitemap https://www.idp.com/mauritius/sitemap.xml
sitemap https://www.idp.com/middleeast/sitemap.xml
sitemap https://www.idp.com/nepal/sitemap.xml
sitemap https://www.idp.com/newzealand/sitemap.xml
sitemap https://www.idp.com/nigeria/sitemap.xml
sitemap https://www.idp.com/oman/sitemap.xml
sitemap https://www.idp.com/pakistan/sitemap.xml
sitemap https://www.idp.com/philippines/sitemap.xml
sitemap https://www.idp.com/saudiarabia/sitemap.xml
sitemap https://www.idp.com/singapore/sitemap.xml
sitemap https://www.idp.com/srilanka/sitemap.xml
sitemap https://www.idp.com/taiwan/sitemap.xml
sitemap https://www.idp.com/thailand/sitemap.xml
sitemap https://www.idp.com/turkey/sitemap.xml
sitemap https://www.idp.com/uae/sitemap.xml
sitemap https://www.idp.com/vietnam/sitemap.xml

Comments

  • Sitewide Directives
  • Country Specific Directives
  • Legacy Directives
  • Disallow directory listing
  • Ignore search query parameters
  • Disallow campaign pages
  • Disallow individual events pages
  • Sitemaps
  • Country Sitemaps
  • Delayed Crawlers
  • Disallowed Crawlers

Warnings

  • 3 invalid lines.