Log In Sign Up

computerweekly.it
robots.txt

Robots Exclusion Standard data for computerweekly.it

Resource Scan

Scan Details

Site Domain computerweekly.it
Base Domain computerweekly.it
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer redirected incorrectly.
Last Scan2025-11-25T11:54:34+00:00
Next Scan 2025-12-25T11:54:34+00:00

Last Successful Scan

Scanned2025-10-26T07:32:08+00:00
URL https://computerweekly.it/robots.txt
Domain IPs 35.214.171.193
Response IP 35.214.171.193
Found Yes
Hash 35ca7555dd08ee9a520198be440f1bfeb8f702ad51fd38412e634922df522652
SimHash 76d00b07c7c0

Groups

gptbot
claudebot
claude-user
claude-searchbot
ccbot
google-extended
applebot-extended
facebookbot
meta-externalagent
meta-externalfetcher
diffbot
perplexitybot
perplexity‑user
omgili
omgilibot
webzio-extended
imagesiftbot
bytespider
tiktokspider
amazonbot
youbot
semrushbot-ocob
petalbot
velenpublicwebcrawler
turnitinbot
timpibot
oai-searchbot
icc-crawler
ai2bot
ai2bot-dolma
dataforseobot
awariobot
awariosmartbot
awariorssbot
google-cloudvertexbot
pangubot
kangaroo bot
sentibot
img2dataset
meltwater
seekr
peer39_crawler
cohere-ai
cohere-training-data-crawler
duckassistbot
scrapy
cotoyogi
aihitbot
factset_spyderbot
firecrawlagent

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Comments

  • Block any non-specified AI crawlers (e.g., new
  • or unknown bots) from using content for training
  • AI models, while allowing the website to be
  • indexed and accessed by bots. These directives
  • are still experimental and may not be supported
  • by all AI crawlers.
Back to top

Warnings

  • `content-usage` is not a known field.
  • `disallowaitraining` is not a known field.
Back to top