Log In Sign Up

jobfin.be
robots.txt

Robots Exclusion Standard data for jobfin.be

Resource Scan

Scan Details

Site Domain jobfin.be
Base Domain jobfin.be
Scan Status Ok
Last Scan2025-02-11T12:19:53+00:00
Next Scan 2025-03-13T12:19:53+00:00

Last Scan

Scanned2025-02-11T12:19:53+00:00
URL https://jobfin.be/robots.txt
Redirect https://www.jobfin.be/robots.txt
Redirect Domain www.jobfin.be
Redirect Base jobfin.be
Domain IPs 104.21.89.210, 172.67.147.89, 2606:4700:3035::ac43:9359, 2606:4700:3036::6815:59d2
Redirect IPs 104.21.89.210, 172.67.147.89, 2606:4700:3035::ac43:9359, 2606:4700:3036::6815:59d2
Response IP 172.67.147.89
Found Yes
Hash abc1531046607a5cd99074dcf7ab1d1a17037c733c101659f245c190e330e9bb
SimHash 14a159d18692

Groups

*

Rule Path Comment
Disallow /page/*/?s= -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-content/uploads/wpforms/ -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to backend login section
Disallow /search/ block access to internal search result pages
Disallow *?s=* block access to internal search result pages
Disallow *?p=* block access to pages for which permalinks fails
Disallow *%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow *?jobtitel=* block access to internal search result pages
Disallow *?locatie=* block access to internal search result pages
Disallow *?diploma=* block access to internal search result pages
Disallow *?dienst=* block access to internal search result pages
Disallow *?contracttype=* block access to internal search result pages
Disallow *%26jobtitel%3D* block access to internal search result pages
Disallow *%26locatie%3D* block access to internal search result pages
Disallow *%26diploma%3D* block access to internal search result pages
Disallow *%26dienst%3D* block access to internal search result pages
Disallow *%26contracttype%3D* block access to internal search result pages

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.jobfin.be/sitemap_index.xml
Back to top

Comments

  • Disallow search query url's
  • Trying to prevent access from AI bots
  • User-agent: ChatGPT-User
  • Disallow: /
  • User-agent: GPTBot
  • Disallow: /
  • Trying to avoid other bots
  • *********
  • ***************
  • ******** ******
  • ******* ******
  • ******
  • ***((((
  • ((((((((
  • (((((((((((((((
  • (((((((((
Back to top