bonsempregos.com
robots.txt

Robots Exclusion Standard data for bonsempregos.com

Resource Scan

Scan Details

Site Domain bonsempregos.com
Base Domain bonsempregos.com
Scan Status Ok
Last Scan2026-01-02T15:49:04+00:00
Next Scan 2026-01-09T15:49:04+00:00

Last Scan

Scanned2026-01-02T15:49:04+00:00
URL https://bonsempregos.com/robots.txt
Redirect https://www.bonsempregos.com/robots.txt
Redirect Domain www.bonsempregos.com
Redirect Base bonsempregos.com
Domain IPs 104.21.23.110, 172.67.210.205, 2606:4700:3030::6815:176e, 2606:4700:3034::ac43:d2cd
Redirect IPs 104.21.23.110, 172.67.210.205, 2606:4700:3030::6815:176e, 2606:4700:3034::ac43:d2cd
Response IP 104.21.23.110
Found Yes
Hash 93b51c2665b70201d83d02f46337d2ea96a6e02decce68ff846141e5e52cb900
SimHash 6d50d8500ef1

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /login
Disallow /register
Disallow /password/
Disallow */fraud-flagged
Allow /storage/
Allow /build/

Other Records

Field Value
sitemap https://www.bonsempregos.com/sitemap.xml

Comments

  • Disallow admin areas
  • Pagination is allowed - duplicate content is handled via canonical URLs
  • Previously blocked: /empregos?*, /empresas?*, /candidatos?*
  • Removed 2025-12-12 to fix Google indexing issue
  • Disallow fraud flagged pages
  • Allow important directories
  • Sitemap location
  • Crawl-delay removed 2025-12-12 to improve indexing speed
  • Google ignores crawl-delay anyway, and Bing respects it which slows indexing