bestjobs.eu
robots.txt

Robots Exclusion Standard data for bestjobs.eu

Resource Scan

Scan Details

Site Domain bestjobs.eu
Base Domain bestjobs.eu
Scan Status Ok
Last Scan2024-10-14T09:42:52+00:00
Next Scan 2024-10-21T09:42:52+00:00

Last Scan

Scanned2024-10-14T09:42:52+00:00
URL https://bestjobs.eu/robots.txt
Redirect https://www.bestjobs.eu/robots.txt
Redirect Domain www.bestjobs.eu
Redirect Base bestjobs.eu
Domain IPs 35.244.184.128
Redirect IPs 35.244.184.128
Response IP 35.244.184.128
Found Yes
Hash 43ed026132b9989e93bf216caa4005525cdd9ff98c65e494b73bcf40c8d1e417
SimHash 5f5dc1505fb3

Groups

*

Rule Path
Disallow

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow /estimated-salary-info/*
Disallow /ro/locuri-de-munca/beneficii/*
Disallow /en/jobs/benefits/*
Disallow /hu/allasok/juttatasok/*

Other Records

Field Value
sitemap https://www.bestjobs.eu/sitemap/sitemap.xml
sitemap https://www.bestjobs.eu/casual/sitemap_index.xml

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449