Log In Sign Up

fatbike.com.pl
robots.txt

Robots Exclusion Standard data for fatbike.com.pl

Resource Scan

Scan Details

Site Domain fatbike.com.pl
Base Domain fatbike.com.pl
Scan Status Ok
Last Scan2024-09-05T15:17:58+00:00
Next Scan 2024-10-05T15:17:58+00:00

Last Scan

Scanned2024-09-05T15:17:58+00:00
URL https://fatbike.com.pl/robots.txt
Domain IPs 85.128.159.195
Response IP 85.128.159.195
Found Yes
Hash 817739cfb037030e7ac33dd8b57acbfe0a67e4b208ee70e1210918c15bd0c348
SimHash 035dc1715631

Groups

*

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/
Disallow /print.php

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 500

googlebot

Rule Path
Disallow

googlebot-news

Rule Path
Disallow

googlebot-video

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

slurp

Rule Path
Disallow

yahoo-blogs

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

facebook

Rule Path
Disallow

googlebot-image

Rule Path
Allow /images/
Disallow /*drukuj*

*

Rule Path
Disallow /add-siccodes/

magpie-crawler

Rule Path
Disallow /

wikido

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

exabot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

wesee:search

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

no_user_agent

Rule Path
Disallow /

bot[\s_ :,\.\;\/\\-]

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.fatbike.com.pl/mapa-strony
Back to top

Comments

  • www.robotstxt.org/
  • disallow all
  • but allow only important bots
  • User-agent: msnbot
  • Disallow:
  • User-agent: msnbot-media
  • Disallow:
  • User-agent: *
  • Allow: /
  • User-agent: Googlebot-Image
  • Allow: /images/
Back to top

Warnings

  • 2 invalid lines.
Back to top