Log In Sign Up

blacksheep.gr
robots.txt

Robots Exclusion Standard data for blacksheep.gr

Resource Scan

Scan Details

Site Domain blacksheep.gr
Base Domain blacksheep.gr
Scan Status Ok
Last Scan2025-10-16T00:35:24+00:00
Next Scan 2025-11-15T00:35:24+00:00

Last Scan

Scanned2025-10-16T00:35:24+00:00
URL https://blacksheep.gr/robots.txt
Domain IPs 104.21.65.94, 172.67.189.185, 2606:4700:3031::ac43:bdb9, 2606:4700:3036::6815:415e
Response IP 104.21.65.94
Found Yes
Hash 272b7b9d616d8cd72e6ec229ae8a5752293fb1f622072dc1754be7f58a83d76e
SimHash 251e5841e671

Groups

*

Rule Path
Allow /assets/css/
Allow /assets/js/
Allow /assets/images/
Allow /assets/dist/
Disallow /admin/
Disallow /includes/
Disallow /scripts/
Disallow /config/
Disallow /vendor/
Disallow /logs/
Disallow /*?search=
Disallow /*?filter=
Disallow /*?sort=
Disallow /*?page=
Disallow /*?q=
Disallow /search
Disallow /search/
Disallow /user/
Disallow /account/
Disallow /profile/
Disallow /cart/
Disallow /checkout/
Disallow /orders/
Disallow /wishlist/
Disallow /api/
Disallow /ajax/
Disallow /test/
Disallow /tmp/
Disallow /temp/
Disallow /*.php$
Disallow /*.log$
Disallow /*.txt$
Disallow /*.inc$
Allow /
Allow /product/
Allow /category/
Allow /about
Allow /contact
Allow /faq
Allow /terms-of-service
Allow /privacy-policy
Allow /categories
Allow /collection

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://blacksheep.gr/sitemap.xml
Back to top

Comments

  • Robots.txt for BlackSheep.gr
  • Generated on: 2025-01-20
  • Allow all crawlers by default
  • Allow important assets for proper page rendering
  • Block admin and system directories
  • Block search and filter URLs to prevent duplicate content
  • Block user-specific pages
  • Block API endpoints if any
  • Block temporary or test pages
  • Block file extensions that shouldn't be indexed
  • Allow important pages (explicit allows for clarity)
  • Sitemaps
  • Special instructions for major search engines
  • Google
  • Allow everything by default (inherits from *)
  • Bing
  • Facebook (for social sharing previews)
  • Twitter (for social sharing previews)
  • Block problematic bots (optional - add as needed)
Back to top