Log In Sign Up

cwjobs.co.uk
robots.txt

Robots Exclusion Standard data for cwjobs.co.uk

Resource Scan

Scan Details

Site Domain cwjobs.co.uk
Base Domain cwjobs.co.uk
Scan Status Ok
Last Scan2024-10-05T00:09:27+00:00
Next Scan 2024-10-12T00:09:27+00:00

Last Scan

Scanned2024-10-05T00:09:27+00:00
URL https://www.cwjobs.co.uk/robots.txt
Domain IPs 125.252.218.150
Response IP 125.252.218.150
Found Yes
Hash 1c6f4dd546f74a06559a36a381a6f4f3428df88509e3203c9c246edddc151dca
SimHash a6e3aba8f1e2

Groups

amazonbot

Rule Path
Disallow /

twitterbot
adsbot-google
adsbot-google-mobile
mediapartners-google

Rule Path Comment
Allow /$ -
Allow /*?WT.mc_id= allow aggregated jobs
Allow /*%26WT.mc_id%3D allow aggregated jobs - added 28 01 2020 AC
Allow /*DCMP%3D -
Allow /*campaign-id%3D -
Allow /job/ -
Disallow /job/*? -
Allow /jobs/ -
Allow /jobs/*?q= -
Disallow /jobs/*?q=*& -
Disallow /jobs/*? -
Allow /jobs-at/ -
Disallow /jobs-at/*? -
Allow /advice/ -
Disallow /advice/*? -
Allow /salary-checker/ -
Disallow /salary-checker/*? -
Allow /recruiters/ RD 10/3/2020
Disallow /recruiters/*? -
Allow /recruiters-products/ RD 10/3/2020
Disallow /recruiters-products/*? -
Allow /.well-known/ RD 25/8/22
Disallow / -
Allow /sharedcontent/img/ RD 13/2/24

applebot
bingbot
bingpreview
msnbot
slurp

Rule Path Comment
Allow /$ -
Allow /JobSearch/ -
Allow /job/ -
Disallow /job/*? -
Allow /jobs/ -
Allow /jobs/*?q= -
Disallow /jobs/*?q=*& -
Disallow /jobs/*? -
Allow /jobs-at/ -
Disallow /jobs-at/*? -
Allow /advice/ -
Disallow /advice/*? -
Allow /salary-checker/ -
Disallow /salary-checker/*? -
Allow /recruiter-products/ -
Disallow /recruiter-products/*? -
Allow /insidejob/ -
Disallow /insidejob/*? -
Allow /courses/ -
Disallow /courses/*? -
Allow /.well-known/ RD 25/8/22
Disallow / -

*

Rule Path Comment
Allow /*?*&crawl=true$ -
Allow /*?WT.mc_id=A_RE_GOOORG -
Allow /*%26WT.mc_id%3DA_RE_GOOORG added 28/01/2020
Allow /*WT.mc_id%3DA_SE_Go added 28/01/2020
Disallow /*WT.mc_id%3D -
Allow /*?q*&page* -
Disallow /*%3D%3D$ -
Disallow /*.ashx$ -
Disallow /*entryUrl%3D* 5/10/18 RD
Disallow /*entryurl%3D* 5/10/18 RD
Disallow /job/*Visitor-Source%3D* 5/10/18 RD
Disallow /*?page=1$ don't index page=1
Disallow /*%26page%3D* this is where there is a page parameter on a facited link - its a catch all
Disallow /*?WT.mc_id=* stop aggregated jobs
Disallow /*%26WT.mc_id%3D* stop aggregated jobs
Disallow /*?similarjobsmodal=* -
Disallow /*%26similarjobsmodal%3D* -
Disallow /jsd/ -
Disallow /insidejob/wp-admin/ -
Disallow */apply$ -
Disallow */apply?* -
Disallow /Login/ -
Disallow /savedjobs -
Disallow /personalisation -
Disallow /*?CompanyId=* -
Disallow /Accessible/ -
Disallow /Authenticated/ -
Disallow /cannedsearch/ -
Disallow /CannedSearch/ -
Disallow /CompanySearch/ -
Disallow /JobSearch/JobBasket.aspx -
Disallow /JobSearch/JobContactDetails.aspx -
Disallow /JobSearch/JobPrinterFriendlyDetails.aspx -
Disallow /JobSearch/jobs-on-a-map.aspx -
Disallow /JobSearch/PromotedClick.aspx -
Disallow /JobSearch/RSS.aspx -
Disallow /JobSearch/SendToAFriend.aspx -
Disallow /JobSearch/UnAuthApplyOnline.aspx -
Disallow /JobSearch/PreExternalApplyOnline.aspx -
Disallow /JobSearch/JobsByEmailSetup.aspx -
Disallow /JobLink/ -
Disallow /pgl/ -
Disallow /pjb_ui/ -
Disallow /WebServices/ -
Disallow /Webservices/ -
Disallow /Maintenance.aspx -
Disallow /service/nolayout.aspx -
Disallow /CompanyBrowse/Roger-Jones-Recruitment-Ltd_Vacancies_c336468.html -
Disallow /*%26Rate%3D* -
Disallow /*%26RateType%3D* -
Disallow /*%26ValidFromDay%3D* -
Disallow /*%26CompanyType%3D* -
Disallow /*companytype%3D* -
Disallow /*%26Discipline%3D* -
Disallow /*%26JobType1%3D* -
Disallow /*%26JobType%3D* -
Disallow /*%26radius%3D* this is where the radius is in a secondary faceted link
Disallow /*%26Radius%3D* this is where the radius is in a secondary faceted link
Disallow /*%26postedwithin%3D* this is where the postedwithin is in a secondary faceted link
Disallow /*%26Postedwithin%3D* this is where the postedwithin is in a secondary faceted link
Disallow /*%26sort%3D* this is where the sort is in a secondary faceted link
Disallow /*%26Sort%3D* this is where the sort is in a secondary faceted link
Disallow /*%26salary%3D* this is where the salary is in a secondary faceted link
Disallow /*%26Salary%3D* this is where the salary is in a secondary faceted link
Disallow /*/jbe/* -
Disallow /JobsByEmail/* -
Disallow /JobSearch/AdvancedJobSearch.aspx -
Disallow /account/ updated 09/03/17 RD
Disallow /Account/ updated 09/03/17 RD
Disallow /AccountDetails/* -
Disallow /*/similarjobs/* -
Disallow /Jobsearch/ApplyViaDocuments.aspx -
Disallow /~/* -
Disallow /login -
Disallow /Help/* -
Disallow /SalaryChecker/SalaryCheckerResults.aspx -
Disallow /jobseekers/*_new.asp -
Disallow /tag/* -
Disallow /jobwidget/* -
Disallow /JobSearch/LocationSelect.aspx -
Disallow /JobSearch/EmailLink.aspx -
Disallow /JobSearch/*/api/* -
Disallow /jobsearch/*/api/* -
Disallow /help/ -
Disallow */articledetail.asp -
Disallow */archivedetails.asp -
Disallow /LocationHub/ -
Disallow /jobs-at/jobs/in- -
Disallow /jobs-at/*/jobs/part-time/ -
Disallow /jobs-at/*/jobs/permanent/ -
Disallow /jobs-at/*/jobs/contract/ -
Disallow /jobs-at/*/jobs/temporary/ -
Disallow /mail_rd.asp -
Disallow /newwave_rd.asp -
Disallow /sitecore/*/www. -
Disallow /careers-advice/*/www. -
Disallow /jobs-at/*/jobs/work-from-home/ -
Disallow /*?action= -
Disallow /*%26action%3D -
Disallow /jobs/work-from-home/*/in-* -
Disallow /mya/ -
Disallow /optimizely-edge/ -
Disallow /analytics/analytics-library.js -
Disallow /application/ -
Back to top

Comments

  • cwjobs - last edited 08 Oct 2018
  • 8/9/2017
  • 2/11/2016
  • 25/08/2015 RD added to block facets on unified results
  • 17/08/2016 Added by RD to block call backs
  • 05/07/2016 Added by RD to block call backs - considered and decided to block!
  • 22/05/2015 added to block facets on results.aspx
  • Disallow: /*&JobTitle=*
  • to be added 08/09/2015
  • added 18/05/2016 to stop Google logging these
  • added 18/05/2016 other old URLs to block RD
  • 6/09/21 RD
  • 26/7/22 RD
  • 21/01/23 JM
  • 18/12/23 RD
  • 24/04/24 RD - removed 10/05/24 JM
  • Disallow: /jobs/*/in-*?*cmp=*
Back to top