Log In Sign Up

totaljobs.com
robots.txt

Robots Exclusion Standard data for totaljobs.com

Resource Scan

Scan Details

Site Domain totaljobs.com
Base Domain totaljobs.com
Scan Status Ok
Last Scan2024-11-13T02:50:09+00:00
Next Scan 2024-11-20T02:50:09+00:00

Last Scan

Scanned2024-11-13T02:50:09+00:00
URL https://www.totaljobs.com/robots.txt
Domain IPs 125.252.218.150
Response IP 125.252.218.150
Found Yes
Hash d39609eaf319520a29dd4e80d6f1aa1281fd080fea0330bc8ed9ab1489cc3818
SimHash f7e192ad63a2

Groups

amazonbot

Rule Path
Disallow /

twitterbot

Rule Path Comment
Allow /$ -
Allow /*?WT.mc_id= allow aggregated jobs
Allow /*%26WT.mc_id%3D allow aggregated jobs - added 28 01 2020 AC
Allow /*DCMP%3D -
Allow /*campaign-id%3D -
Allow /job/ -
Disallow /job/*? -
Allow /jobs/ -
Allow /jobs/*?q= -
Allow /jobs/*?cmp=* added RD 13/5/24
Disallow /jobs/*?q=*& -
Disallow /jobs/*? -
Allow /jobs-at/ -
Disallow /jobs-at/*? -
Allow /advice/ -
Disallow /advice/*? -
Allow /salary-checker/ -
Allow /salary-planner* RD 8/4/22
Disallow /salary-checker/*? -
Allow /insidejob/ -
Disallow /insidejob/*? -
Allow /recruiters/ RD 10/3/2020
Allow /recruiters -
Allow /recruiter-advice/ -
Allow /.well-known/ RD 25/8/22
Disallow / -
Allow /sharedcontent/img/ RD 13/2/24

applebot
bingbot
bingpreview
msnbot
slurp

Rule Path Comment
Allow /$ -
Allow /JobSearch/ -
Allow /job/ -
Disallow /job/*? -
Allow /jobs/ -
Allow /jobs/*?q= -
Disallow /jobs/*?q=*& -
Disallow /jobs/*? -
Allow /jobs-at/ -
Disallow /jobs-at/*? -
Allow /advice/ -
Disallow /advice/*? -
Allow /salary-checker/ -
Disallow /salary-checker/*? -
Allow /recruiters/ -
Disallow /recruiters/*? -
Allow /insidejob/ -
Disallow /insidejob/*? -
Allow /courses/ -
Disallow /courses/*? -
Disallow / -
Allow /.well-known/ RD 25/8/22

*

Rule Path Comment
Allow /*?*&crawl=true$ -
Allow /*WT.mc_id%3DA_RE_GOOORG* -
Allow /*WT.mc_id%3DA_SE_Go* added 28/01/2020
Allow /*WT.mc_id%3DR_AE_* added 05/05/2020
Allow /*WT.mc_id%3DB2B-* added 06/05/2021 RD
Allow /*WT.mc_id%3Ddvideo added 15/02/23 JM
Disallow /*WT.mc_id%3D -
Disallow /courses/search-results? -
Disallow /CompanySearch/CompanyLogoLink.asp -
Disallow /CompanySearch/CompanyVacancies.aspx -
Disallow /LearnDirect/ -
Disallow /*w%3D*%26as%3D -
Disallow /*DCMP%3D -
Disallow /*campaign-id%3D -
Disallow /*utm_medium%3D -
Disallow /browse-jobs?redir=*&CompanyTerm= -
Disallow /browse-jobs?redir=*&Keywords=& -
Disallow /*/main.asp -
Disallow /entry -
Disallow /*Visitor-Source%3D -
Allow /*?q*&page= 8/9/2017
Disallow /*%3D%3D$ 2/11/2016
Disallow /*.ashx$ -
Disallow /*%26page%3D this is where there is a page parameter on a facetted link - its a catch all
Disallow /*similarjobsmodal%3D -
Disallow /*entryUrl%3D 5/10/18 RD
Disallow /*entryurl%3D 5/10/18 RD
Disallow /job/*Visitor-Source%3D 5/10/18 RD
Disallow /jsd/ -
Disallow /insidejob/wp-admin/ -
Disallow /*/apply? -
Disallow /*/apply$ -
Disallow /Login/ -
Disallow /savedjobs -
Disallow /personalisation/ -
Disallow /Accessible/ -
Disallow /Authenticated/ -
Disallow /authenticated/ -
Disallow /cannedsearch/ -
Disallow /CannedSearch/ -
Disallow /CompanySearch/CompanyProfileFrameset.aspx -
Disallow /*/JobSearch/JobContactDetails.aspx -
Disallow /*/JobSearch/JobPrinterFriendlyDetails.aspx -
Disallow /*/JobSearch/Results.aspx -
Disallow /*/JobSearch/SendToAFriend.aspx -
Disallow /*/JobSearch/UnAuthApplyOnline.aspx -
Disallow /*Broaden%3D -
Disallow /*Rate%3D -
Disallow /*RateType%3D -
Disallow /*ValidFromDay%3D -
Disallow /*CompanyType%3D -
Disallow /*JobTitle%3D -
Disallow /*Discipline%3D -
Disallow /*JobType1%3D -
Disallow /*JobType%3D -
Disallow /*Radius%3D -
Disallow /*companytype%3D 5/10/18 RD
Disallow /*%26radius%3D -
Disallow /*?radius= -
Disallow /*industry%3D 2/12/20 RD
Disallow /*Industry%3D -
Disallow /*salary%3D -
Disallow /*Salary%3D -
Disallow /*jobtype%3D -
Disallow /*postedwithin%3D -
Disallow /*PostedWithin%3D -
Disallow /*duration%3D 9/4/19 RD
Disallow /JobSearch/JobBasket.aspx -
Disallow /JobSearch/JobContactDetails.aspx -
Disallow /JobSearch/JobPrinterFriendlyDetails.aspx -
Disallow /JobSearch/jobs-on-a-map.aspx -
Disallow /JobSearch/PromotedClick.aspx -
Disallow /JobSearch/SendToAFriend.aspx -
Disallow /JobSearch/UnAuthApplyOnline.aspx -
Disallow /JobSearch/PreExternalApplyOnline.aspx -
Disallow /JobSearch/JobsByEmailSetup.aspx -
Disallow /Jobsearch/Results.aspx -
Disallow /JobSearch/*/api/ -
Disallow /jobsearch/*/api/ -
Disallow /help/ -
Disallow */articledetail.asp -
Disallow */archivedetails.asp -
Disallow /LocationHub/ -
Disallow /jobs-at/jobs/in- -
Disallow /jobs-at/*/jobs/part-time/ -
Disallow /jobs-at/*/jobs/permanent/ -
Disallow /jobs-at/*/jobs/contract/ -
Disallow /jobs-at/*/jobs/temporary/ -
Disallow /mail_rd.asp -
Disallow /newwave_rd.asp -
Disallow /sitecore/*/www. -
Disallow /careers-advice/*/www. -
Disallow /JobSearch/LocationSelect.aspx -
Disallow /JobSearch/EmailLink.aspx -
Disallow /JobSearch/JobMailRecruiter.aspx -
Disallow /*.swf$ -
Disallow /*/jbe/ -
Disallow /JobsByEmail/ -
Disallow /JobSearch/AdvancedJobSearch.aspx -
Disallow /account/ updated 01/02/15 RD
Disallow /Account/ updated 17/07/16 RD
Disallow /*/similarjobs/* -
Disallow /Jobsearch/ApplyViaDocuments.aspx -
Disallow /~/ -
Disallow /login/ -
Disallow /Help/ -
Disallow /SalaryChecker/SalaryCheckerResults.aspx -
Disallow /jobseekers/*_new.asp -
Disallow /tag/ -
Disallow /jobwidget/ -
Disallow /advertise- -
Disallow /JobSearch/RSS.aspx -
Disallow /MSN/ -
Disallow /pgl/ -
Disallow /pjb_ui/ -
Disallow /WebServices/ -
Disallow /Webservices/ -
Disallow /Maintenance.aspx -
Disallow /service/nolayout.aspx -
Disallow /jobs-at/*/jobs/work-from-home/ -
Disallow /harm/ -
Disallow /*?action= -
Disallow /*%26action%3D -
Disallow /jobs/work-from-home/*/in-* -
Disallow /recruiters/campaign-* -
Disallow /*?*wt=* 5/8/24 RD
Disallow /membersarea/ -
Disallow /membersarea -
Disallow /headerfooter/ -
Disallow /mya/ -
Disallow /optimizely-edge/ -
Disallow /analytics/analytics-library.js -
Disallow /application/ -
Disallow /jad/ 5/8/24 RD
Disallow /searchbar/ 20/8/24 RD
Back to top

Comments

  • tj.com - last edited 08 04 2022 RD
  • User-agent: AdsBot-Google
  • User-agent: AdsBot-Google-Mobile
  • User-agent: Mediapartners-Google
  • Disallow: /recruiters/*? # added 06/05/2021 RD
  • Disallow: /jobs/*&r= # 5/10/18 RD
  • Disallow: /jobs/*?r= # 5/10/18 RD
  • 2/12/20 RD
  • 13/06/22 RD
  • 26/7/22 RD
  • 21/01/23 JM
  • 18/12/23 RD
  • 24/04/24 RD - removed 10/05/24 JM
  • Disallow: /jobs/*/in-*?*cmp=*
Back to top