Log In Sign Up

worktree.ca
robots.txt

Robots Exclusion Standard data for worktree.ca

Resource Scan

Scan Details

Site Domain worktree.ca
Base Domain worktree.ca
Scan Status Ok
Last Scan2025-11-17T09:06:06+00:00
Next Scan 2025-12-01T09:06:06+00:00

Last Scan

Scanned2025-11-17T09:06:06+00:00
URL https://worktree.ca/robots.txt
Domain IPs 2607:5300:401:3200::d, 51.222.19.172
Response IP 51.222.19.172
Found Yes
Hash 3fafb699852fe6a994c73d1de002701b11dd8904efdce7d78b133bef111c5a60
SimHash f6d15a00c7c0

Groups

gptbot
claudebot
claude-web
ccbot
googlebot-extended
applebot-extended
facebookbot
meta-externalagent
meta-externalfetcher
diffbot
perplexitybot
omgili
omgilibot
webzio-extended
imagesiftbot
bytespider
amazonbot
youbot
semrushbot-ocob
petalbot
velenpublicwebcrawler
turnitinbot
timpibot
oai-searchbot
icc-crawler
ai2bot
ai2bot-dolma
dataforseobot
awariobot
awariosmartbot
awariorssbot
google-cloudvertexbot
pangubot
kangaroo bot
sentibot
img2dataset
meltwater
seekr
peer39_crawler
cohere-ai
cohere-training-data-crawler
duckassistbot
scrapy

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Back to top

Comments

  • Block all known AI crawlers and assistants
  • from using content for training AI models.
  • Source: https://robotstxt.com/ai
  • Block any non-specified AI crawlers (e.g., new
  • or unknown bots) from using content for training
  • AI models. This directive is still experimental
  • and may not be supported by all AI crawlers.
Back to top

Warnings

  • `disallowaitraining` is not a known field.
Back to top