Log In Sign Up

jobhunterparadise.com
robots.txt

Robots Exclusion Standard data for jobhunterparadise.com

Resource Scan

Scan Details

Site Domain jobhunterparadise.com
Base Domain jobhunterparadise.com
Scan Status Ok
Last Scan2026-01-03T18:57:10+00:00
Next Scan 2026-01-10T18:57:10+00:00

Last Scan

Scanned2026-01-03T18:57:10+00:00
URL https://jobhunterparadise.com/robots.txt
Domain IPs 104.21.82.132, 172.67.158.57, 2606:4700:3030::6815:5284, 2606:4700:3037::ac43:9e39
Response IP 104.21.82.132
Found Yes
Hash 1d6cc790ad6970f5c547a3bc4f24dd6fe0b311bf0fdf6cc9733533ec5bcbc960
SimHash 6010f950e9a4

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /
Disallow /api/
Disallow /admin/
Disallow /_astro/
Disallow /dev/
Disallow /*.json$
Allow /jobs/
Allow /about/
Allow /search/
Back to top

Other Records

Field Value
sitemap https://jobhunterparadise.com/sitemap.xml
Back to top

Comments

  • High-traffic crawlers
  • Block AI training crawlers (optional)
  • Block certain paths
  • Allow important directories
  • Sitemap location
Back to top