Log In Sign Up

blog.max-fun.de
robots.txt

Robots Exclusion Standard data for blog.max-fun.de

Resource Scan

Scan Details

Site Domain blog.max-fun.de
Base Domain max-fun.de
Scan Status Ok
Last Scan2024-10-04T04:50:06+00:00
Next Scan 2024-10-05T04:50:06+00:00

Last Scan

Scanned2024-10-04T04:50:06+00:00
URL https://blog.max-fun.de/robots.txt
Domain IPs 185.3.235.206
Response IP 185.3.235.206
Found Yes
Hash 10ffb852007517db18e1040cb38082ee61706c8c1c4836686a8ee40e7d16baee
SimHash 682a5bfd0cc4

Groups

linkwalker

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

slurp/2.0

Rule Path
Disallow /

slurp.so/1.0

Rule Path
Disallow /

slurp/si

Rule Path
Disallow /

slurp/cat

Rule Path
Disallow /

slurp

Rule Path
Disallow /

wget

Rule Path
Disallow /

fast-webcrawler/3.8

Rule Path
Disallow /

googlebot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

chatgpt-user

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /
Back to top

Comments

  • robots.txt for http://blog.max-fun.de/
  • $Id: robots.txt,v 1.17 2004/11/01 02:37:04 renaudb Exp $
  • spammer
  • company letting crawl the community: nono
  • I do not like Microsoft, please keep out
  • Inktomisearch
  • Inktomisearch
  • Inktomisearch
  • Inktomiesearch
  • Inktomiesearch
  • Wget
  • Overture
  • Dark Visitors robots.txt
  • AI Assistant
  • https://darkvisitors.com/agents/chatgpt-user
  • AI Assistant
  • https://darkvisitors.com/agents/cohere-ai
  • AI Data Scraper
  • https://darkvisitors.com/agents/anthropic-ai
  • AI Data Scraper
  • https://darkvisitors.com/agents/ccbot
  • AI Data Scraper
  • https://darkvisitors.com/agents/facebookbot
  • AI Data Scraper
  • https://darkvisitors.com/agents/google-extended
  • AI Data Scraper
  • https://darkvisitors.com/agents/gptbot
  • AI Data Scraper
  • https://darkvisitors.com/agents/omgilibot
Back to top