Log In Sign Up

getwork.com
robots.txt

Robots Exclusion Standard data for getwork.com

Resource Scan

Scan Details

Site Domain getwork.com
Base Domain getwork.com
Scan Status Ok
Last Scan2026-01-23T16:53:06+00:00
Next Scan 2026-02-06T16:53:06+00:00

Last Scan

Scanned2026-01-23T16:53:06+00:00
URL https://getwork.com/robots.txt
Redirect https://www.getwork.com/robots.txt
Redirect Domain www.getwork.com
Redirect Base getwork.com
Domain IPs 2600:1f14:49a:a300:4d07:fd22:9235:6b41, 2600:1f14:49a:a301:8ac4:20af:51b5:edcc, 2600:1f14:49a:a302:d31e:ef8:64f7:d94e, 35.84.47.59, 52.25.192.160, 52.35.238.157
Redirect IPs 2600:1f14:49a:a300:4d07:fd22:9235:6b41, 2600:1f14:49a:a301:8ac4:20af:51b5:edcc, 2600:1f14:49a:a302:d31e:ef8:64f7:d94e, 35.84.47.59, 52.25.192.160, 52.35.238.157
Response IP 52.35.238.157
Found Yes
Hash e693bb611dbc1a60e2d00f7dbd747a064e1adfb6a8200ad06e09b211e5fef77c
SimHash 601009508d64

Groups

amazonbot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /search?
Disallow /jobs/search?
Disallow /to-rent?
Disallow /for-sale?
Disallow /goto/ad/
Disallow /jobs/goto/ad/
Disallow /land/ad/
Disallow /jobs/land/ad/
Disallow /advanced-search?
Disallow /jobs/advanced-search?
Disallow /jobs/my-alerts?
Disallow /my-alerts?
Disallow /jobiak/
Disallow /get_avg?
Disallow /get_stats?

adsbot-google
adsbot-google-mobile

Rule Path
Disallow /create_notification
Disallow /jobs/create_notification

ccbot
gptbot
chatgpt-user
google-extended
bytespider
diffbot
facebookbot
omgili
applebot-extended
perplexitybot
amazonbot
claudebot
omgilibot
anthropic-ai
claude-web
imagesiftbot
youbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.getwork.com/sitemap_index_details.jobs_GW.xml
Back to top

Comments

  • TODO: actually think properly about what needs to be here for
  • Getwork on the Adzuna stack (this was copied across from
  • adzuna.com)
  • Disallow /create_notification endpoint from being accessed by the AdsBot
  • https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers
  • Sitemap links
  • JOB-2438: disallow ChatGPT crawlers (see https://darkvisitors.com/agents)
Back to top