Log In Sign Up

android.izzysoft.de
robots.txt

Robots Exclusion Standard data for android.izzysoft.de

Resource Scan

Scan Details

Site Domain android.izzysoft.de
Base Domain izzysoft.de
Scan Status Ok
Last Scan2025-09-23T11:05:23+00:00
Next Scan 2025-10-07T11:05:23+00:00

Last Scan

Scanned2025-09-23T11:05:23+00:00
URL https://android.izzysoft.de/robots.txt
Domain IPs 144.76.109.58
Response IP 144.76.109.58
Found Yes
Hash cb0f078441e3b2b8e046b78061fb31bd5bf22ba918c7d0d73221458ac2da8836
SimHash 60709553c753

Groups

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

dataprovider.com

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

*

Rule Path
Disallow /images
Disallow /hpbots
Disallow /repo
Disallow /*.apk$

Other Records

Field Value
crawl-delay 5

googlebot

Rule Path
Allow /images

googlebot-mobile

Rule Path
Allow /images

googlebot-image

Rule Path
Disallow /images

jobboersebot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://android.izzysoft.de/sitemap.xml
Back to top

Comments

  • --=[ SEO Bots ]=--
  • SEO Bot https://ahrefs.com/robot/ https://darkvisitors.com/agents/ahrefsbot
  • https://darkvisitors.com/agents/ahrefssiteaudit
  • https://www.babbar.tech/crawler SEO bot; used to provide a set of online marketing and referencing tools for the SEO community.
  • https://darkvisitors.com/agents/rogerbot
  • https://darkvisitors.com/agents/zoombot for online marketing campaigns
  • --=[ AI Bots used to feed LLMs ]=--
  • https://darkvisitors.com/agents/bytespider allegedly used to download training data for its LLMs
  • https://darkvisitors.com/agents/anthropic-ai
  • https://darkvisitors.com/agents/diffbot
  • https://darkvisitors.com/agents/facebookbot
  • https://darkvisitors.com/agents/google-extended
  • https://darkvisitors.com/agents/gptbot
  • https://darkvisitors.com/agents/omgili
  • https://darkvisitors.com/agents/meta-externalagent
  • --=[ Scrapers ]=--
  • https://darkvisitors.com/agents/dataprovider-com (paid service)
  • --=[ Unclassified ]=--
  • permit bots only 1 page per 5s (supported by msnbot, Yahoo, Yandex):
  • Bots shall not download stuff
Back to top