Log In Sign Up

myfanbase.de
robots.txt

Robots Exclusion Standard data for myfanbase.de

Resource Scan

Scan Details

Site Domain myfanbase.de
Base Domain myfanbase.de
Scan Status Ok
Last Scan2024-06-27T02:50:15+00:00
Next Scan 2024-07-04T02:50:15+00:00

Last Scan

Scanned2024-06-27T02:50:15+00:00
URL https://myfanbase.de/robots.txt
Redirect https://www.myfanbase.de/robots.txt
Redirect Domain www.myfanbase.de
Redirect Base myfanbase.de
Domain IPs 85.13.145.138
Redirect IPs 85.13.145.138
Response IP 85.13.145.138
Found Yes
Hash fc66ad415a2b066149b607c46339b9cd23a53b36e9ddbccba178a880f55d2765
SimHash 341033805037

Groups

*

Rule Path
Disallow /lib/
Disallow /images/
Disallow /grafik/
Disallow /videos/
Disallow /admin/

twitterbot

Rule Path
Disallow *
Allow /pics/

*

Rule Path
Disallow *mode%3D

*

Rule Path
Disallow *allowembed%3D

proximic

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

applebot-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /
Back to top

Comments

  • Robots.txt file from http://www.myfanbase.de
  • Disallow: /pics/
  • Exclude all pages with mode parameter to be crawled
  • e.g. myfanbase.de/?mode=mobile
  • e.g. myfanbase.de/?allowembed=true
  • Request Bots to delay crawling
  • Comscore proximic marketing (delay 3 minutes)
  • Disable Crawlers to use content for AI training
  • Apple Intelligence Bot
  • OpenAI Bot
  • Google Bard, Vertex AI Bots
Back to top