dotnetcore.show
robots.txt

Robots Exclusion Standard data for dotnetcore.show

Resource Scan

Scan Details

Site Domain dotnetcore.show
Base Domain dotnetcore.show
Scan Status Ok
Last Scan2024-10-22T07:49:01+00:00
Next Scan 2024-11-21T07:49:01+00:00

Last Scan

Scanned2024-10-22T07:49:01+00:00
URL https://dotnetcore.show/robots.txt
Domain IPs 13.251.96.10, 2406:da18:880:3800::c8, 2406:da18:880:3801::c8, 52.74.166.77
Response IP 18.139.194.139
Found Yes
Hash 4f5f5d3aa99f3fa34da18789fdf22c6e5ced45cb3a9134abaa3f8e966a094fb2
SimHash 74901bd1e6b4

Groups

*

Rule Path
Disallow /categories/
Disallow /tags/
Allow /
Disallow /categories/*
Disallow /tags/*

adsbot-google
amazonbot
anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
diffbot
facebookbot
friendlycrawler
google-extended
googleother
gptbot
img2dataset
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
perplexitybot
youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://dotnetcore.show/sitemap.xml

Comments

  • crawler for all LLMs - disallow everything
  • based on https://github.com/ai-robots-txt/ai.robots.txt