Log In Sign Up

milkround.com
robots.txt

Robots Exclusion Standard data for milkround.com

Resource Scan

Scan Details

Site Domain milkround.com
Base Domain milkround.com
Scan Status Ok
Last Scan2024-11-16T21:29:12+00:00
Next Scan 2024-11-23T21:29:12+00:00

Last Scan

Scanned2024-11-16T21:29:12+00:00
URL https://www.milkround.com/robots.txt
Domain IPs 125.252.218.150
Response IP 125.252.218.150
Found Yes
Hash 8c713dd6b0caae86de09447a7fc70f17fc9825d3d276b95d9fb3a3f46f517229
SimHash ee319aa86372

Groups

amazonbot

Rule Path
Disallow /

twitterbot
adsbot-google
adsbot-google-mobile
mediapartners-google
botify

Rule Path Comment
Allow /$ -
Allow /*?WT.mc_id= allow aggregated jobs
Allow /*%26WT.mc_id%3D allow aggregated jobs - added 28 01 2020 AC
Allow /*DCMP%3D -
Allow /*campaign-id%3D -
Allow /job/ -
Disallow /job/*? -
Allow /jobs/ -
Allow /jobs/*?q= -
Disallow /jobs/*?q=*& -
Disallow /jobs/*? -
Allow /jobs-at/ -
Disallow /jobs-at/*? -
Allow /advice/ -
Disallow /advice/*? -
Allow /salary-checker/ -
Disallow /salary-checker/*? -
Allow /insidejob/ -
Disallow /insidejob/*? -
Allow /recruiters/ RD 10/3/2020
Disallow /recruiters/*? -
Allow /recruiters-products/ RD 10/3/2020
Disallow /recruiters-products/*? -
Disallow / -

applebot
bingbot
bingbot
yahoo! slurp
slurp

Rule Path
Allow /$
Allow /JobSearch/
Allow /job/
Disallow /job/*?
Allow /jobs/
Allow /jobs/*?q=
Disallow /jobs/*?q=*&
Disallow /jobs/*?
Allow /jobs-at/
Disallow /jobs-at/*?
Allow /advice/
Disallow /advice/*?
Allow /salary-checker/
Disallow /salary-checker/*?
Allow /recruiter-products/
Disallow /recruiter-products/*?
Allow /insidejob/
Disallow /insidejob/*?
Allow /courses/
Disallow /courses/*?
Disallow /

Other Records

Field Value
crawl-delay 5

*

Rule Path Comment
Disallow /*%3D%3D$ -
Disallow /*.ashx$ -
Allow /*?*&crawl=true$ -
Allow /*WT.mc_id%3DA_RE_GOOORG -
Allow /*WT.mc_id%3DA_SE_Go added 28/01/2020
Disallow /*WT.mc_id%3D -
Disallow /courses/search-results? -
Disallow /CompanySearch/CompanyLogoLink.asp -
Disallow /CompanySearch/CompanyVacancies.aspx -
Disallow /LearnDirect/ -
Disallow /*w%3D*%26as%3D -
Disallow /*campaign-id%3D -
Disallow /*utm_medium%3D -
Disallow /browse-jobs?redir=*&CompanyTerm= -
Disallow /browse-jobs?redir=*&Keywords=& -
Disallow /*/main.asp -
Disallow /entry -
Disallow /*Visitor-Source%3D -
Allow /*?q*&page= 8/9/2017
Disallow /*%3D%3D$ 2/11/2016
Disallow /*.ashx$ -
Disallow /*%26page%3D this is where there is a page parameter on a facited link - its a catch all
Disallow /*similarjobsmodal%3D -
Disallow /*entryUrl%3D 5/10/18 RD
Disallow /*entryurl%3D 5/10/18 RD
Disallow /job/*Visitor-Source%3D 5/10/18 RD
Disallow /jsd/ -
Disallow /insidejob/wp-admin/ -
Disallow /*/apply? -
Disallow /*/apply$ -
Disallow /Login/ -
Disallow /savedjobs -
Disallow /personalisation/ -
Disallow /Accessible/ -
Disallow /Authenticated/ -
Disallow /authenticated/ -
Disallow /cannedsearch/ -
Disallow /CannedSearch/ -
Disallow /CompanySearch/CompanyProfileFrameset.aspx -
Disallow /*/JobSearch/JobContactDetails.aspx -
Disallow /*/JobSearch/JobPrinterFriendlyDetails.aspx -
Disallow /*/JobSearch/Results.aspx -
Disallow /*/JobSearch/SendToAFriend.aspx -
Disallow /*/JobSearch/UnAuthApplyOnline.aspx -
Disallow /*Broaden%3D -
Disallow /*Rate%3D -
Disallow /*RateType%3D -
Disallow /*ValidFromDay%3D -
Disallow /*CompanyType%3D -
Disallow /*JobTitle%3D -
Disallow /*Discipline%3D -
Disallow /*JobType1%3D -
Disallow /*JobType%3D -
Disallow /*Radius%3D -
Disallow /*companytype%3D 5/10/18 RD
Disallow /*%26salary%3D* -
Disallow /*%26Salary%3D* -
Disallow /*?salary=* -
Disallow /*?Salary=* -
Disallow /*salarytypeid%3D* -
Disallow /*SalaryTypeid%3D* -
Disallow /*companytype%3D* -
Disallow /*postedwithin%3D* -
Disallow /*?Sort=* -
Disallow /*?sort=* -
Disallow /*%26Sort%3D* -
Disallow /*%26sort%3D* -
Disallow /*?radius=* -
Disallow /*?Radius=* -
Disallow /*%26radius%3D* -
Disallow /*%26Radius%3D* -
Disallow /*%26r%3D* -
Disallow /*?r=* -
Disallow /*%26page%3D* this is where there is a page parameter on a facited link - its a catch all
Disallow /jsd/ -
Disallow /Login/ -
Disallow /savedjobs -
Disallow /*?CompanyId=* -
Disallow /Accessible/ -
Disallow /Authenticated/ -
Disallow /authenticated/ -
Disallow /cannedsearch/ -
Disallow /CannedSearch/ -
Disallow /JobSearch/JobContactDetails.aspx -
Disallow /JobSearch/JobPrinterFriendlyDetails.aspx -
Disallow /JobSearch/jobs-on-a-map.aspx -
Disallow /JobSearch/PromotedClick.aspx -
Disallow /JobSearch/SendToAFriend.aspx -
Disallow /JobSearch/UnAuthApplyOnline.aspx -
Disallow /JobSearch/JobBasket.aspx -
Disallow /JobSearch/PreExternalApplyOnline.aspx -
Disallow /JobSearch/JobsByEmailSetup.aspx -
Disallow /pgl/ -
Disallow /pjb_ui/ -
Disallow /JobSearch/RSS.aspx -
Disallow /WebServices/ -
Disallow /Webservices/ -
Disallow /Maintenance.aspx -
Disallow /service/nolayout.aspx -
Disallow /*%26Rate%3D* -
Disallow /*%26RateType%3D* -
Disallow /*%26ValidFromDay%3D* -
Disallow /*%26CompanyType%3D* -
Disallow /*%26Discipline%3D* -
Disallow /*%26JobType1%3D* -
Disallow /*%26Radius%3D20* -
Disallow /*%26Radius%3D30* -
Disallow /*%26Radius%3D5* -
Disallow /*/jbe/* -
Disallow /JobsByEmail/ -
Disallow /JobSearch/AdvancedJobSearch.aspx -
Disallow /account/ updated 01/02/15 RD
Disallow /Account/ updated 17/07/16 RD
Disallow /*/similarjobs/* -
Disallow /Jobsearch/ApplyViaDocuments.aspx -
Disallow /~/* -
Disallow /login -
Disallow /Help/ -
Disallow /SalaryChecker/SalaryCheckerResults.aspx -
Disallow /jobseekers/*_new.asp -
Disallow /tag/ -
Disallow /jobwidget/ -
Disallow /JobSearch/LocationSelect.aspx -
Disallow /JobSearch/EmailLink.aspx -
Disallow /JobSearch/*/api/* -
Disallow /jobsearch/*/api/* -
Disallow /help/ -
Disallow */articledetail.asp -
Disallow */archivedetails.asp -
Disallow /LocationHub/ -
Disallow /jobs-at/jobs/in- -
Disallow /jobs-at/*/jobs/part-time/ -
Disallow /jobs-at/*/jobs/permanent/ -
Disallow /jobs-at/*/jobs/contract/ -
Disallow /jobs-at/*/jobs/temporary/ -
Disallow /mail_rd.asp -
Disallow /newwave_rd.asp -
Disallow /sitecore/*/www. -
Disallow /careers-advice/*/www. -
Disallow /jobs-at/*/jobs/work-from-home/ -
Disallow /*?action= -
Disallow /*%26action%3D -
Disallow /jobs/work-from-home/*/in-* -
Disallow /membersarea/ -
Disallow /membersarea -
Disallow /headerfooter/ -
Disallow /mya/ -
Disallow /optimizely-edge/ -
Disallow /analytics/analytics-library.js -
Disallow /application/ -
Back to top

Comments

  • milkround.com - last edited 12 June 2019
  • 2/11/2016
  • 12/06/2019 RD
  • 25/08/2015 RD added to block facets on unified results
  • 05/07/2016 Added by RD to block call backs - considered and decied to allow!
  • 05/07/2016 Added by RD to block call backs - considered and decided to block!
  • 22/05/2015 added to block facets on results.aspx
  • Disallow: /*&JobTitle=* removed 1/2/16 RD
  • to be added 08/09/2015
  • added 18/05/2016 to stop Google logging these
  • added 18/05/2016 other old URLs to block RD
  • 6/09/21 RD
  • 13/06/22 RD
  • 26/7/22 RD
  • 21/01/23 JM
  • 18/12/23 RD
  • 24/04/24 RD - removed 10/05/24 JM
  • Disallow: /jobs/*/in-*?*cmp=*
Back to top