Log In Sign Up

caterer.com
robots.txt

Robots Exclusion Standard data for caterer.com

Resource Scan

Scan Details

Site Domain caterer.com
Base Domain caterer.com
Scan Status Ok
Last Scan2024-11-10T19:46:07+00:00
Next Scan 2024-11-17T19:46:07+00:00

Last Scan

Scanned2024-11-10T19:46:07+00:00
URL https://www.caterer.com/robots.txt
Domain IPs 125.252.218.150
Response IP 125.252.218.150
Found Yes
Hash cf98efb609e2824d4ce4652f46bc947131f60ec56c2a3357059d37212bc655a1
SimHash a7f19a00f3e2

Groups

amazonbot

Rule Path
Disallow /

twitterbot

Rule Path Comment
Allow /$ -
Allow /*?WT.mc_id= allow aggregated jobs
Allow /*%26WT.mc_id%3D allow aggregated jobs - added 28 01 2020 AC
Allow /*DCMP%3D -
Allow /*campaign-id%3D -
Allow /job/ -
Disallow /job/*? -
Allow /jobs/ -
Allow /jobs/*?q= -
Allow /jobs/*?cmp=* added RD 13/5/24
Disallow /jobs/*?q=*& -
Disallow /jobs/*? -
Allow /jobs-at/ -
Disallow /jobs-at/*? -
Allow /advice/ -
Disallow /advice/*? -
Allow /salary-checker/ -
Disallow /salary-checker/*? -
Allow /recruiters/ RD 10/3/2020
Allow /recruiters-products/ RD 10/3/2020
Disallow /recruiters-products/*? -
Allow /recruiters RP 29/04/2021
Allow /recruiter-products/ AC 16/08/2022
Disallow / -
Allow /.well-known/ RD 25/8/22
Allow /account/signin -
Allow /account/register -
Allow /sharedcontent/img/ RD 13/2/24

applebot
bingbot
bingpreview
msnbot
slurp

Rule Path Comment
Allow /$ -
Allow /JobSearch/ -
Allow /job/ -
Disallow /job/*? -
Allow /jobs/ -
Allow /jobs/*?q= -
Disallow /jobs/*?q=*& -
Disallow /jobs/*? -
Allow /jobs-at/ -
Disallow /jobs-at/*? -
Allow /advice/ -
Disallow /advice/*? -
Allow /salary-checker/ -
Disallow /salary-checker/*? -
Allow /recruiter-products/ -
Disallow /recruiter-products/*? -
Allow /insidejob/ -
Disallow /insidejob/*? -
Allow /courses/ -
Disallow /courses/*? -
Disallow / -
Allow /.well-known/ RD 25/8/22
Allow /account/ updated 30/08/22 RD

*

Rule Path Comment
Allow /*?*&crawl=true$ -
Allow /*WT.mc_id%3DA_RE_GOOORG* -
Allow /*WT.mc_id%3DA_SE_Go* added 28/01/2020
Allow /*WT.mc_id%3DR_AE_* added 05/05/2020
Allow /*WT.mc_id%3DB2B-* added 06/05/2021 RD
Disallow /*WT.mc_id%3D -
Allow /*?q*&page=* -
Disallow /*%3D%3D$ -
Disallow /*.ashx$ -
Disallow /jobs-at/hospitality-search-international/* -
Disallow /*entryUrl%3D* 5/10/18 RD
Disallow /*entryurl%3D* 5/10/18 RD
Disallow /job/*Visitor-Source%3D* 5/10/18 RD
Disallow /*salary%3D* -
Disallow /*Salary%3D* -
Disallow /*salarytypeid%3D* -
Disallow /*SalaryTypeid%3D* -
Disallow /*companytype%3D* -
Disallow /*postedwithin%3D* -
Disallow /*Sort%3D* -
Disallow /*sort%3D* -
Disallow /*radius%3D* -
Disallow /*Radius%3D* -
Disallow /*%26q%3D* -
Disallow /jobs/*%26r%3D* RD 8/9/2017
Disallow /*?r=* -
Disallow /*?page=1$ don't index page=1
Disallow /*%26page%3D* this is where there is a page parameter on a facited link - its a catch all
Disallow /*?similarjobsmodal=* -
Disallow /*%26similarjobsmodal%3D* -
Disallow /jsd/ -
Disallow /jsd/ -
Disallow /insidejob/wp-admin/ -
Disallow /*/apply?* -
Disallow /*/apply$ -
Disallow /Login/ -
Disallow /savedjobs -
Disallow /*?CompanyId=* -
Disallow /Accessible/ -
Disallow /Authenticated/ -
Disallow /authenticated/ -
Disallow /cannedsearch/ -
Disallow /CannedSearch/ -
Disallow /CompanySearch/ -
Disallow /JobSearch/JobContactDetails.aspx -
Disallow /JobSearch/JobPrinterFriendlyDetails.aspx -
Disallow /JobSearch/jobs-on-a-map.aspx -
Disallow /JobSearch/PromotedClick.aspx -
Disallow /JobSearch/SendToAFriend.aspx -
Disallow /JobSearch/UnAuthApplyOnline.aspx -
Disallow /JobSearch/JobBasket.aspx -
Disallow /JobSearch/PreExternalApplyOnline.aspx -
Disallow /JobSearch/JobsByEmailSetup.aspx -
Disallow /pgl/ -
Disallow /pjb_ui/ -
Disallow /JobSearch/RSS.aspx -
Disallow /WebServices/ -
Disallow /Webservices/ -
Disallow /Maintenance.aspx -
Disallow /service/nolayout.aspx -
Disallow /*%26Rate%3D* -
Disallow /*%26RateType%3D* -
Disallow /*%26ValidFromDay%3D* -
Disallow /*%26CompanyType%3D* -
Disallow /*%26Discipline%3D* -
Disallow /*%26JobType1%3D* -
Disallow /*%26JobType%3D* -
Disallow /*%26Radius%3D20* -
Disallow /*%26Radius%3D30* -
Disallow /*%26Radius%3D5* -
Disallow /*?Rate=* -
Disallow /*?RateType=* -
Disallow /*?ValidFromDay=* -
Disallow /*?CompanyType=* -
Disallow /*?Discipline=* -
Disallow /*?JobType1=* -
Disallow /*?JobType=* -
Disallow /*?Radius=20* -
Disallow /*?Radius=30* -
Disallow /*?Radius=5* -
Disallow /*/jbe/* -
Disallow /JobsByEmail/ -
Disallow /JobSearch/AdvancedJobSearch.aspx -
Disallow /account/ updated 01/02/15 RD
Disallow /Account/ updated 17/07/16 RD
Allow /account/signin -
Allow /account/register -
Disallow /*/similarjobs/* -
Disallow /Jobsearch/ApplyViaDocuments.aspx -
Disallow /~/* -
Disallow /login -
Disallow /Help/ -
Disallow /SalaryChecker/SalaryCheckerResults.aspx -
Disallow /jobseekers/*_new.asp -
Disallow /tag/ -
Disallow /jobwidget/ -
Disallow /JobSearch/LocationSelect.aspx -
Disallow /JobSearch/EmailLink.aspx -
Disallow /JobSearch/*/api/* -
Disallow /jobsearch/*/api/* -
Disallow /help/ -
Disallow */articledetail.asp -
Disallow */archivedetails.asp -
Disallow /LocationHub/ -
Disallow /jobs-at/jobs/in- -
Disallow /jobs-at/*/jobs/part-time/ -
Disallow /jobs-at/*/jobs/permanent/ -
Disallow /jobs-at/*/jobs/contract/ -
Disallow /jobs-at/*/jobs/temporary/ -
Disallow /mail_rd.asp -
Disallow /newwave_rd.asp -
Disallow /sitecore/*/www. -
Disallow /careers-advice/*/www. -
Disallow /jobs-at/*/jobs/work-from-home/ -
Disallow /*?action= -
Disallow /*%26action%3D -
Disallow /jobs/work-from-home/*/in-* -
Disallow /membersarea/ -
Disallow /membersarea -
Disallow /headerfooter/ -
Disallow /mya/ -
Disallow /optimizely-edge/ -
Disallow /analytics/analytics-library.js -
Disallow /application/ -
Back to top

Comments

  • caterer.com - last edited 5 Oct 2018 RD
  • User-agent: AdsBot-Google
  • User-agent: AdsBot-Google-Mobile
  • User-agent: Mediapartners-Google
  • Disallow: /recruiters/*? # added 06/05/2021 RD
  • RD 30/8/22
  • 8/9/2017
  • 2/11/2016
  • specific clients
  • 25/08/2015 RD added to block facets on unified results
  • Disallow: /*?q=* # RD 8/9/2017
  • 05/07/2016 Added by RD to block call backs - considered and decied to allow!
  • 17/08/2016 Added by RD to block call backs
  • 22/05/2015 added to block facets on results.aspx
  • Disallow: /*?JobTitle=* removed 1/2/16 RD
  • to be added 08/09/2015
  • RD 30/8/22
  • added 18/05/2016 to stop Google logging these
  • added 18/05/2016 other old URLs to block RD
  • 6/09/21 RD
  • 13/06/22 RD
  • 26/7/22 RD
  • 21/01/23 JM
  • 18/12/23 RD
  • 24/04/24 RD - removed 10/05/24 JM
  • Disallow: /jobs/*/in-*?*cmp=*
Back to top

Warnings

  • 1 invalid line.
Back to top