hays.com
robots.txt

Robots Exclusion Standard data for hays.com

Resource Scan

Scan Details

Site Domain hays.com
Base Domain hays.com
Scan Status Ok
Last Scan2025-02-23T12:43:26+00:00
Next Scan 2025-03-25T12:43:26+00:00

Last Scan

Scanned2025-02-23T12:43:26+00:00
URL https://hays.com/robots.txt
Redirect https://www.hays.com/robots.txt
Redirect Domain www.hays.com
Redirect Base hays.com
Domain IPs 45.60.33.111, 45.60.39.111
Redirect IPs 45.60.35.111
Response IP 45.60.35.111
Found Yes
Hash 85efea238ce2bc5a32a21fd42d648fee463c93d667dc5c9003f7cdc52aa1ea4b
SimHash 495a9d97ccb8

Groups

*

Rule Path
Disallow /search?
Disallow /it/search?
Disallow /*.pdf
Disallow /page-not-found
Disallow /c/portal/
Disallow /en/c/portal/
Disallow /_Incapsula_Resource?
Disallow /combo?
Disallow /jobs-search/
Disallow /viewemailink/
Disallow /online-training/employee/
Disallow /online-training/employee/*
Disallow /clientPortal/clientDashboard/
Disallow /DocExSignIn/beginSigningSession
Disallow /timesheetAlert/reviewTimesheet/
Disallow /timesheetAlert/approveTimesheet/
Disallow /gdprThankYou/
Disallow /job-search*
Disallow /jobs-search*
Disallow /en/
Disallow /en/web/*/
Disallow /en-US/web/hays-usa/
Disallow /job-detail/*/apply
Disallow /job-detail/*/job-apply-confirmation
Disallow /portal/auth
Disallow /career-advice/search
Disallow /market-insights/search
Disallow /account/online-learning/training?
Disallow /online-learning*?*
Disallow /featured-employers*?*
Disallow /account/login
Disallow /account/saved-jobs
Disallow /my-account/unsubscribe
Disallow /my-account/unsubscribe-email
Disallow /my-account/unsubscribe-email-confirmed
Disallow /my-account/update-preferences-email
Disallow /my-account/sent-marketing-preferences-email
Disallow /my-account/marketing-preferences
Disallow /my-account/marketing-preferences-confirmed
Disallow /account/update-your-details
Disallow /account/update-data-privacy
Disallow /my-account/profile
Disallow /my-account/job-alerts
Disallow /my-account/create-job-alert
Disallow /my-account/create-job-alert-email
Disallow /my-account/job-alert-confirmation
Disallow /account/delete-job-alert
Disallow /my-account/manage-resume
Disallow /my-account/change-email
Disallow /my-account/change-password
Disallow /my-account/update-whatsapp
Disallow /my-account/update-whatsapp-number-email
Disallow /my-account/whatsapp-number-confirmed?submit=Y
Disallow /account/reset
Disallow /account/error
Disallow /recruitment/life-sciences
Disallow /market-insights/reports/thank-you
Disallow /recruitment/contact-us/thank-you
Disallow /jobs/foreign-job-seekers
Disallow /Job/Detail/*
Disallow *?*p_p_id
Disallow /web/hays-salary-guide-ae
Disallow /web/hays-salary-guide-ae/saudi-arabia-salary-guide
Disallow /web/hays-salary-guide-ae/emiratisation-salary-guide
Allow /o/hays-web-branding-theme/images/favicon.ico
Disallow /o/
Disallow /job-archive
Disallow /recaptcha-form

fidget-spinner-bot

Rule Path
Disallow /

python-requests/2.21.0

Rule Path
Disallow /

sogou web spider/4.0; +http://www.sogou.com/docs/help/webmasters.htm

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

omniexplorer_bot

Rule Path
Disallow /

yahooseeker/cafekelsa

Rule Path
Disallow /

wijubot/1.0

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

seoscanners.net/1

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

siteexplorer/1.0

Rule Path
Disallow /

siteexplorer/1.0b

Rule Path
Disallow /

curious george - www.analyticsseo.com/crawler

Rule Path
Disallow /

spbot

Rule Path
Disallow /

mozilla/5.0 (compatible; domainappender /1.0; +http://www.profound.net/domainappender)

Rule Path
Disallow /

sitecheck-sitecrawl by siteimprove.com

Rule Path
Disallow /

feedly/1.0

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gigabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

twiceler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

jobcrawlerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

innovantagebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

innovantagebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.hays.com/sitemap.xml
sitemap https://www.hays.com/sitemap/en-US/job-sitemap.xml
sitemap https://www.hays.com/blog-sitemap.xml