careerstructure.com
robots.txt
Robots Exclusion Standard data for careerstructure.com
Resource Scan
Scan Details
| Site Domain | careerstructure.com |
| Base Domain | careerstructure.com |
| Scan Status | Ok |
| Last Scan | 2024-11-17T01:32:43+00:00 |
| Next Scan | 2024-11-24T01:32:43+00:00 |
Last Scan
| Scanned | 2024-11-17T01:32:43+00:00 |
| URL | https://www.careerstructure.com/robots.txt |
| Domain IPs | 125.252.218.150 |
| Response IP | 125.252.218.150 |
| Found | Yes |
| Hash | 7352eb06d61c32621900a948b95796853d32c0a69e125dba8709ad183477a448 |
| SimHash | 3ff38a21f1f2 |
Groups
twitterbot
adsbot-google
adsbot-google-mobile
mediapartners-google
| Rule | Path | Comment |
|---|---|---|
| Allow | /$ | - |
| Allow | /*?WT.mc_id= | allow aggregated jobs |
| Allow | /*%26WT.mc_id%3D | allow aggregated jobs - added 28 01 2020 AC |
| Allow | /*DCMP%3D | - |
| Allow | /*campaign-id%3D | - |
| Allow | /job/ | - |
| Disallow | /job/*? | - |
| Allow | /jobs/ | - |
| Allow | /jobs/*?q= | - |
| Disallow | /jobs/*?q=*& | - |
| Disallow | /jobs/*? | - |
| Allow | /jobs-at/ | - |
| Disallow | /jobs-at/*? | - |
| Allow | /advice/ | - |
| Disallow | /advice/*? | - |
| Allow | /salary-checker/ | - |
| Disallow | /salary-checker/*? | - |
| Allow | /insidejob/ | - |
| Disallow | /insidejob/*? | - |
| Allow | /recruiters/ | RD 10/3/2020 |
| Disallow | /recruiters/*? | - |
| Allow | /recruiters-products/ | RD 10/3/2020 |
| Disallow | /recruiters-products/*? | - |
| Allow | /.well-known/ | RD 25/8/22 |
| Disallow | / | - |
| Allow | /sharedcontent/img/ | RD 13/2/24 |
applebot
bingbot
bingpreview
msnbot
slurp
| Rule | Path | Comment |
|---|---|---|
| Allow | /$ | - |
| Allow | /JobSearch/ | - |
| Allow | /job/ | - |
| Disallow | /job/*? | - |
| Allow | /jobs/ | - |
| Allow | /jobs/*?q= | - |
| Disallow | /jobs/*?q=*& | - |
| Disallow | /jobs/*? | - |
| Allow | /jobs-at/ | - |
| Disallow | /jobs-at/*? | - |
| Allow | /advice/ | - |
| Disallow | /advice/*? | - |
| Allow | /salary-checker/ | - |
| Disallow | /salary-checker/*? | - |
| Allow | /recruiters/ | - |
| Disallow | /recruiters/*? | - |
| Allow | /insidejob/ | - |
| Disallow | /insidejob/*? | - |
| Allow | /courses/ | - |
| Disallow | /courses/*? | - |
| Allow | /.well-known/ | RD 25/8/22 |
| Disallow | / | - |
*
| Rule | Path | Comment |
|---|---|---|
| Allow | /*?WT.mc_id= | allow aggregated jobs |
| Allow | /*%26WT.mc_id%3D | allow aggregated jobs - added 28 01 2020 AC |
| Allow | /*DCMP%3D | - |
| Allow | /*campaign-id%3D | - |
| Allow | /*?*&crawl=true$ | - |
| Allow | /*?WT.mc_id=A_RE_GOOORG | - |
| Allow | /*%26WT.mc_id%3DA_RE_GOOORG | added 28/01/2020 |
| Allow | /*WT.mc_id%3DA_SE_Go | added 28/01/2020 |
| Disallow | /*WT.mc_id%3D | - |
| Disallow | /constructionspace/ | - |
| Disallow | /*%3D%3D$ | - |
| Disallow | /*?page=1$ | don't index page=17 |
| Disallow | /*%26page%3D* | this is where there is a page parameter on a facited link - its a catch all |
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /jsd/ | - |
| Disallow | /insidejob/wp-admin/ | - |
| Disallow | /*/apply?* | - |
| Disallow | /*/apply$ | - |
| Disallow | /Login/ | - |
| Disallow | /savedjobs | - |
| Disallow | /personalisation | - |
| Disallow | /*%26Rate%3D* | - |
| Disallow | /*%26RateType%3D* | - |
| Disallow | /*%26ValidFromDay%3D* | - |
| Disallow | /*%26CompanyType%3D* | - |
| Disallow | /*%26Discipline%3D* | - |
| Disallow | /*%26JobType1%3D* | - |
| Disallow | /*%26Radius%3D20* | - |
| Disallow | /*%26Radius%3D30* | - |
| Disallow | /*%26Radius%3D5* | - |
| Disallow | /*?CompanyId=* | - |
| Disallow | /Accessible/ | - |
| Disallow | /Authenticated/ | - |
| Disallow | /cannedsearch/ | - |
| Disallow | /CannedSearch/ | - |
| Disallow | /CompanySearch/ | - |
| Disallow | /JobSearch/JobContactDetails.aspx | - |
| Disallow | /JobSearch/JobPrinterFriendlyDetails.aspx | - |
| Disallow | /JobSearch/jobs-on-a-map.aspx | - |
| Disallow | /JobSearch/PromotedClick.aspx | - |
| Disallow | /JobSearch/SendToAFriend.aspx | - |
| Disallow | /JobSearch/UnAuthApplyOnline.aspx | - |
| Disallow | /JobSearch/JobBasket.aspx | - |
| Disallow | /JobSearch/PreExternalApplyOnline.aspx | - |
| Disallow | /JobSearch/JobsByEmailSetup.aspx | - |
| Disallow | /pgl/ | - |
| Disallow | /pjb_ui/ | - |
| Disallow | /JobSearch/RSS.aspx | - |
| Disallow | /WebServices/ | - |
| Disallow | /Webservices/ | - |
| Disallow | /Maintenance.aspx | - |
| Disallow | /service/nolayout.aspx | - |
| Disallow | /*/jbe/* | - |
| Disallow | /JobsByEmail/ | - |
| Disallow | /JobSearch/AdvancedJobSearch.aspx | - |
| Disallow | /account/ | updated 01/02/15 RD |
| Disallow | /Account/ | updated 17/07/16 RD |
| Disallow | /*/similarjobs/* | - |
| Disallow | /Jobsearch/ApplyViaDocuments.aspx | - |
| Disallow | /~/* | - |
| Disallow | /login | - |
| Disallow | /Help/ | - |
| Disallow | /SalaryChecker/SalaryCheckerResults.aspx | - |
| Disallow | /jobseekers/*_new.asp | - |
| Disallow | /tag/ | - |
| Disallow | /jobwidget/ | - |
| Disallow | /JobSearch/LocationSelect.aspx | - |
| Disallow | /JobSearch/EmailLink.aspx | - |
| Disallow | /JobSearch/*/api/* | - |
| Disallow | /jobsearch/*/api/* | - |
| Disallow | /help/ | - |
| Disallow | */articledetail.asp | - |
| Disallow | */archivedetails.asp | - |
| Disallow | /LocationHub/ | - |
| Disallow | /jobs-at/jobs/in- | - |
| Disallow | /jobs-at/*/jobs/part-time/ | - |
| Disallow | /jobs-at/*/jobs/permanent/ | - |
| Disallow | /jobs-at/*/jobs/contract/ | - |
| Disallow | /jobs-at/*/jobs/temporary/ | - |
| Disallow | /mail_rd.asp | - |
| Disallow | /newwave_rd.asp | - |
| Disallow | /sitecore/*/www. | - |
| Disallow | /careers-advice/*/www. | - |
| Disallow | /jobs-at/*/jobs/work-from-home/ | - |
| Disallow | /*?action= | - |
| Disallow | /*%26action%3D | - |
| Disallow | /jobs/work-from-home/*/in-* | - |
| Disallow | /membersarea/ | - |
| Disallow | /membersarea | - |
| Disallow | /headerfooter/ | - |
| Disallow | /mya/ | - |
| Disallow | /optimizely-edge/ | - |
| Disallow | /analytics/analytics-library.js | - |
| Disallow | /application/ | - |
Comments