trabajo.org
robots.txt

Robots Exclusion Standard data for trabajo.org

Resource Scan

Scan Details

Site Domain trabajo.org
Base Domain trabajo.org
Scan Status Ok
Last Scan2026-03-03T00:51:11+00:00
Next Scan 2026-03-10T00:51:11+00:00

Last Scan

Scanned2026-03-03T00:51:11+00:00
URL https://www.trabajo.org/robots.txt
Domain IPs 3.160.188.21, 3.160.188.51, 3.160.188.62, 3.160.188.90
Response IP 3.169.71.26
Found Yes
Hash da9b0a5d5ad9655105d0636cd0a3974347622920072770c2a3f19c71cc9a2958
SimHash 781889d1e5d1

Groups

gptbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

google-extended

Rule Path
Allow /

googleother

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

youbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

diffbot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

imagesiftbot

Rule Path
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

semrushbot

Rule Path
Disallow /ajax/
Disallow /premium/

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /ajax/
Disallow /premium/

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /ajax/
Disallow /premium/

Other Records

Field Value
crawl-delay 10

*

Rule Path
Allow /css/
Allow /js/
Allow /fonts/
Allow /img/
Allow /libraries/bootstrap/
Disallow /njob/
Disallow /ajax/
Disallow /redirect/
Disallow /publish/
Disallow /premium/checkout/
Disallow /alerts/
Disallow /backend/
Disallow /backendv2/
Allow /

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://sitemaps.trabajo.org/index.xml
sitemap https://sitemaps.trabajo.org/www/index.xml

Comments

  • AI & LLM Crawlers - full unrestricted access
  • Search engines - full access
  • SEO crawlers - rate limited
  • Default rules
  • LLM documentation
  • https://www.trabajo.org/llms.txt
  • https://www.trabajo.org/llms-full.txt