Log In Sign Up

cv-library.co.uk
robots.txt

Robots Exclusion Standard data for cv-library.co.uk

Resource Scan

Scan Details

Site Domain cv-library.co.uk
Base Domain cv-library.co.uk
Scan Status Ok
Last Scan2024-05-24T22:54:49+00:00
Next Scan 2024-05-31T22:54:49+00:00

Last Scan

Scanned2024-05-24T22:54:49+00:00
URL https://cv-library.co.uk/robots.txt
Redirect https://www.cv-library.co.uk/robots.txt
Redirect Domain www.cv-library.co.uk
Redirect Base cv-library.co.uk
Domain IPs 109.169.5.15
Redirect IPs 104.18.22.207, 104.18.23.207
Response IP 104.18.22.207
Found Yes
Hash d9662f64fa6112c6ae80ab80fc005374fd4579ff16abc2f211cfea7b9de433e7
SimHash a30dd1458545

Groups

*

Rule Path Comment
Allow / added 30.09.2019
Disallow /*?order* added 02.06.2020
Disallow /*%26order* added 02.06.2020
Disallow /*?salary* added 02.06.2020
Disallow /*%26salary* added 02.06.2020
Disallow /*?perpage* added 02.06.2020
Disallow /*%26perpage* added 02.06.2020
Disallow /*advertise-jobs? added 22.10.2019
Disallow /aff/* -
Disallow /alerts/check-existence -
Disallow /apply/ added 18.12.2019
Disallow /*?ar= added 22.10.2019
Disallow /candidate/ -
Disallow /cgi-bin/ updated 30.09.2019
Disallow /candidate/savejob* -
Disallow /codetrial/ -
Allow /companies/employers added 09.10.2019
Allow /companies/agencies added 09.10.2019
Disallow /companies/employers/by-industry/ added 26.02.2020
Disallow /companies/employers/by-county/ added 26.02.2020
Disallow /companies/agencies/by-industry/ added 26.02.2020
Disallow /companies/agencies/by-county/ added 26.02.2020
Disallow /companies/agencies? added 22.10.2019
Disallow /*cvdb7n%3D added 22.10.2019
Disallow /*cvdb14n%3D added 18.12.2019
Disallow /*?distance=* added 18.12.2019
Disallow /*%26distance%3D* -
Disallow /*?posted=* added 04.06.2020
Disallow /*%26posted%3D* added 04.06.2020
Disallow /domain-spellcheck -
Disallow /employer_trial/ -
Disallow /entercode/ -
Disallow /external/ added 18.12.2019
Disallow /*featured%3D* added 30.09.2019
Allow /job/ added 30.09.2019
Disallow /job/*/*/sanitise -
Disallow /job/*/*/apply -
Disallow /job/*/*/apply/external -
Disallow /job/*/*/email -
Allow /jobs/ added 30.09.2019
Disallow /jobs/*?id=* added 30.09.2019
Disallow /*/key/ added 04/11/2019
Disallow /login/ -
Disallow /*?hl= added 18.12.2019
Disallow /r/ -
Disallow /recruiter-profile/ added 22.10.2019
Disallow /refer/ added 22.10.2019
Disallow /register?* added 22.10.2019
Disallow /register?id=* added 09.10.2019
Disallow /register/*?id=* added 09.10.2019
Disallow /refer/ added 09.10.2019
Disallow /recruiter/ -
Disallow /trial/ -
Disallow /trial-code/ -
Disallow /trialcode/ -
Disallow /*?ts= added 22.10.2019
Disallow /*/*/page/*/ added 04/11/2019
Disallow /press/* -
Disallow /*preview_id%3D added 22.10.2019
Disallow /*search-cvs? added 22.10.2019
Disallow /*search?q= added 22.10.2019
Disallow /*sid%3D added 30.09.2019
Disallow /*?sj= added 22.10.2019
Disallow /*?Subject= added 18.12.2019
Disallow */?utm_source=* added 09.10.2019 / Updated 18.12.2019
Disallow /welcome-back?id=* added 09.10.2019

mediapartners-google
adsbot-google-mobile
adsbot-google

Rule Path
Disallow /alerts/check-existence
Disallow /job/*/*/sanitise
Disallow /collect$

applebot
bingbot
twitterbot

Rule Path Comment
Allow / -
Disallow /*?order* added 02.06.2020
Disallow /*%26order* added 02.06.2020
Disallow /*?salary* added 02.06.2020
Disallow /*%26salary* added 02.06.2020
Disallow /*?perpage* added 02.06.2020
Disallow /*%26perpage* added 02.06.2020
Disallow /ads.txt -
Disallow /*advertise-jobs? added 22.10.2019
Disallow /aff/* -
Disallow /apply/ added 18.12.2019
Disallow /*?ar= added 22.10.2019
Disallow /candidate/ -
Disallow /cgi-bin/ -
Disallow /candidate/savejob* -
Disallow /codetrial/ -
Disallow /alerts/check-existence -
Allow /companies/employers added 09.10.2019
Allow /companies/agencies added 09.10.2019
Disallow /companies/employers/ added 09.10.2019
Disallow /companies/agencies/ added 09.10.2019
Disallow /companies/agencies? added 22.10.2019
Disallow /*cvdb7n%3D added 22.10.2019
Disallow /*cvdb14n%3D added 18.12.2019
Disallow /*?distance=* added 18.12.2019
Disallow /*%26distance%3D* -
Disallow /*?posted=* added 04.06.2020
Disallow /*%26posted%3D* added 04.06.2020
Disallow /domain-spellcheck -
Disallow /employer_trial/ -
Disallow /entercode/ -
Disallow /external/ added 18.12.2019
Disallow /*featured%3D* added 04/11/2019
Disallow /job/*/*/sanitise -
Disallow /job/*/*/apply -
Disallow /job/*/*/apply/external -
Disallow /job/*/*/email -
Allow /jobs/ -
Disallow /jobs/*?id=* added 04/11/2019
Disallow /*/key/ added 04/11/2019
Disallow /login/ -
Disallow /*?hl= added 18.12.2019
Disallow /r/ -
Disallow /recruiter-profile/ added 22.10.2019
Disallow /refer/ added 22.10.2019
Disallow /register?* added 22.10.2019
Disallow /register?id=* added 09.10.2019
Disallow /register/*?id=* added 09.10.2019
Disallow /refer/ added 09.10.2019
Disallow /recruiter/ -
Disallow /trial/ -
Disallow /trial-code/ -
Disallow /trialcode/ -
Disallow /*?ts= added 22.10.2019
Disallow /*/*/page/*/ added 04/11/2019
Disallow /press/* -
Disallow /*preview_id%3D added 22.10.2019
Disallow /search/ added 25.11.19
Disallow /*searchcvs? added 22.10.2019
Disallow /*search?q= added 22.10.2019
Disallow /*sid%3D added 09.10.2019
Disallow /*?sj= added 22.10.2019
Disallow /*?Subject= added 18.12.2019
Disallow */?utm_source=* added 09.10.2019 / Updated 18.12.2019
Disallow /welcome-back?id=* added 09.10.2019

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

slurp

Rule Path
Disallow /

omniexplorer_bot
verticrawlbot
addsearchbot
admantx
ahrefsbot
baiduspider
baiduspider-video
baiduspider-image
changedetection
dotbot
exabot
gozaikbot
grapeshot
ichiro
sogou spider
wesee
wijubot
xovibot
cliqzbot
piplbot
ccbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.cv-library.co.uk/sitemaps/browse-jobs-index.xml
sitemap https://www.cv-library.co.uk/sitemaps/jobs-industry-by-location-index.xml
sitemap https://www.cv-library.co.uk/sitemaps/jobs-industry-index.xml
sitemap https://www.cv-library.co.uk/sitemaps/jobs-location-index.xml
sitemap https://www.cv-library.co.uk/sitemaps/jobs-parent-index.xml
sitemap https://www.cv-library.co.uk/sitemaps/job-posts-index.xml
sitemap https://www.cv-library.co.uk/sitemaps/recruiters-index.xml
sitemap https://www.cv-library.co.uk/sitemaps/salary-guide-index.xml
Back to top

Comments

  • File Generated by CF Worker: Last Updated 12 April 2024
  • removed /ads.txt 30.09.2019
  • removed following Disallows on 23/05/2020
  • /jobs/british-islands*
  • /jobs/england*
  • /search-jobs?*
  • removed following Disallows on 02.06.2020
  • /*?q=* added 18.12.2019
  • The following user-agents ignore the global (*) directives
  • so we need to specify them individually.
  • These disallow directives are non-rendering JS endpoints.
  • added 30.09.2019
  • removed following Disallows on 23/05/2020
  • /jobs/british-islands*
  • /jobs/england*
  • /search-jobs?*
  • removed following Disallows on 02.06.2020
  • /*?q=* added 18.12.2019
  • Baidu Chinese search engine
  • Yandex RU search engine
  • Yahoo
  • Bulk Disallow added 04/11/2019
Back to top