Log In Sign Up

ziprecruiter.com
robots.txt

Robots Exclusion Standard data for ziprecruiter.com

Resource Scan

Scan Details

Site Domain ziprecruiter.com
Base Domain ziprecruiter.com
Scan Status Ok
Last Scan2024-11-12T10:00:19+00:00
Next Scan 2024-11-26T10:00:19+00:00

Last Scan

Scanned2024-11-12T10:00:19+00:00
URL https://ziprecruiter.com/robots.txt
Redirect https://www.ziprecruiter.com/robots.txt
Redirect Domain www.ziprecruiter.com
Redirect Base ziprecruiter.com
Domain IPs 104.16.161.111, 104.16.162.111, 2606:4700::6810:a16f, 2606:4700::6810:a26f
Redirect IPs 104.16.161.111, 104.16.162.111, 2606:4700::6810:a16f, 2606:4700::6810:a26f
Response IP 104.16.161.111
Found Yes
Hash 45c152da511f283080fc317a9ff8281c43e93f474e592f6aa36e0fe89742a263
SimHash 851080644775

Groups

googlebot

Rule Path
Allow /
Disallow /unsubscribe
Disallow /job/track-pageview
Disallow /facebook/
Disallow /amazon/
Disallow /job/track-apply-click
Disallow /job/monster-apply-request
Disallow /?search=
Disallow /geo/job_title
Disallow /blog/wp-admin/
Disallow /contact/zip-resume
Disallow /resume-database/preview-iframe
Disallow /resume/preview/
Disallow /admin
Disallow /login/
Disallow /trk
Disallow /get-auto-banners
Disallow /record-event
Disallow /events
Disallow /apply/
Disallow /eclk/
Disallow /ek/
Disallow /pixel/
Disallow /chat/get-chat-auth-token
Disallow /ajax/
Disallow /tap/
Disallow /jobs/*/closest$
Disallow /jobs/*/clipboard-action$
Disallow /jobs/*/is$
Disallow /svc/seo/
Disallow /submit-reference
Disallow /verify-skill
Disallow /candidate/onboard

bingbot

Rule Path
Allow /
Disallow /unsubscribe
Disallow /job/track-pageview
Disallow /facebook/
Disallow /amazon/
Disallow /job/track-apply-click
Disallow /job/monster-apply-request
Disallow /?search=
Disallow /geo/job_title
Disallow /blog/wp-admin/
Disallow /contact/zip-resume
Disallow /resume-database/preview-iframe
Disallow /resume/preview/
Disallow /admin
Disallow /login/
Disallow /trk
Disallow /get-auto-banners
Disallow /record-event
Disallow /events
Disallow /apply/
Disallow /eclk/
Disallow /ek/
Disallow /pixel/
Disallow /chat/get-chat-auth-token
Disallow /ajax/
Disallow /tap/
Disallow /jobs/*/closest$
Disallow /jobs/*/clipboard-action$
Disallow /jobs/*/is$
Disallow /svc/seo/
Disallow /submit-reference
Disallow /verify-skill
Disallow /jobs-search
Disallow /candidate/onboard

bingbot

Rule Path
Allow /Jobs/
Allow /Salaries/

Other Records

Field Value
crawl-delay 1

google-extended

Rule Path
Allow /
Disallow /unsubscribe
Disallow /job/track-pageview
Disallow /facebook/
Disallow /amazon/
Disallow /job/track-apply-click
Disallow /job/monster-apply-request
Disallow /?search=
Disallow /geo/job_title
Disallow /blog/wp-admin/
Disallow /contact/zip-resume
Disallow /resume-database/preview-iframe
Disallow /resume/preview/
Disallow /admin
Disallow /login/
Disallow /trk
Disallow /get-auto-banners
Disallow /record-event
Disallow /events
Disallow /apply/
Disallow /eclk/
Disallow /ek/
Disallow /pixel/
Disallow /chat/get-chat-auth-token
Disallow /ajax/
Disallow /tap/
Disallow /jobs/*/closest$
Disallow /jobs/*/clipboard-action$
Disallow /jobs/*/is$
Disallow /svc/seo/
Disallow /submit-reference
Disallow /verify-skill
Disallow /jobs-search
Disallow /candidate/onboard

omniexplorer_bot
petalbot
megaindex.ru
scrapy
tweetmemebot
accompanybot
techiejobs.co

Rule Path
Disallow /

*

Rule Path Comment
Allow / -
Disallow /unsubscribe -
Disallow /job/track-pageview -
Disallow /facebook/ -
Disallow /amazon/ -
Disallow /job/track-apply-click -
Disallow /job/monster-apply-request -
Disallow /?search= -
Disallow /geo/job_title -
Disallow /blog/wp-admin/ -
Disallow /contact/zip-resume -
Disallow /resume-database/preview-iframe -
Disallow /resume/preview/ -
Disallow /admin -
Disallow /login/ want the main login page allowed, but not pages under /login/
Disallow /trk -
Disallow /get-auto-banners -
Disallow /record-event -
Disallow /events -
Disallow /apply/ want apply allowed, but not pages under /apply/
Disallow /eclk/ -
Disallow /ek/ -
Disallow /pixel/ -
Disallow /chat/get-chat-auth-token -
Disallow /ajax/ -
Disallow /tap/ -
Disallow /jobs-search -
Disallow /Jobs/ -
Disallow /Salaries/ -
Disallow /co/ -
Disallow /e/ -
Disallow /jobs/*/closest$ -
Disallow /jobs/*/clipboard-action$ -
Disallow /jobs/*/is$ -
Disallow /svc/seo/ -
Disallow /submit-reference -
Disallow /verify-skill -
Back to top

Comments

  • CUSTOM OPT-OUT PER USER AGENT
  • GLOBAL OPT-OUT LIST
  • NOTE: This is the global list, but it is overriden if a user-agent has a custom list defined.
  • Ensure you replicate any new url in those lists if you want to completely block it from being crawled.
  • SERPS are only allowed to be crawled by googlebot
  • DIRPS are only allowed to be crawled by googlebot, bingbot
  • Block URLs that are likely added by js clipboard library
  • Block temporary pages of the go seo app
  • Block profile reference & verify skill links
Back to top

Warnings

  • 1 invalid line.
Back to top