Log In Sign Up

th.jobsdb.com
robots.txt

Robots Exclusion Standard data for th.jobsdb.com

Resource Scan

Scan Details

Site Domain th.jobsdb.com
Base Domain jobsdb.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-05-16T05:26:29+00:00
Next Scan 2025-07-15T05:26:29+00:00

Last Successful Scan

Scanned2025-03-11T04:00:13+00:00
URL https://th.jobsdb.com/robots.txt
Domain IPs 104.18.39.136, 172.64.148.120, 2606:4700:4400::6812:2788, 2606:4700:4400::ac40:9478
Response IP 172.64.148.120
Found Yes
Hash 14078166d0d36fd1b2e612987115a7e150a550b785f332f8eeb54208e523c5e4
SimHash 8206c9c0c4d5

Groups

mediapartners-google
adidxbot

Rule Path
Disallow

*

Rule Path
Disallow */job/
Disallow *?returnUrl=
Disallow *?page=
Disallow /graphql

linkedinbot
baiduspider
petalbot

Rule Path
Disallow /

anthropic-ai
bytespider
ccbot
diffbot
google-extended
omgili
gptbot

Rule Path
Disallow /companies
Disallow */job/

linkedinbot

Rule Path
Allow */job/

facebookexternalhit

Rule Path
Allow */job/*
Allow */jobs*
Allow */*-jobs*
Back to top

Comments

  • robots.txt file for th.jobsdb.com
  • Unrestricted access
  • Default directives
  • Disallowed bots
  • Exceptions
Back to top