Log In Sign Up

fbargainsgalore.co.uk
robots.txt

Robots Exclusion Standard data for fbargainsgalore.co.uk

Resource Scan

Scan Details

Site Domain fbargainsgalore.co.uk
Base Domain fbargainsgalore.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-01T09:11:24+00:00
Next Scan 2025-10-02T09:11:24+00:00

Last Successful Scan

Scanned2025-09-24T05:29:38+00:00
URL https://fbargainsgalore.co.uk/robots.txt
Domain IPs 103.106.229.82, 149.28.136.245, 15.235.181.227, 45.32.123.201
Response IP 149.28.136.245
Found Yes
Hash 71abedf7463a7037ea32656551fc673e541a9bfc34a346e8866d9d9627c7e35f
SimHash 7d5edf51eadb

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-content/uploads/wpo/
Disallow /cart/
Disallow /checkout/
Disallow /customer/
Disallow /account/
Disallow /my-account/
Disallow /login
Disallow /register
Disallow /password-reset
Disallow /wishlist/
Disallow /compare/
Disallow /track-order/
Disallow /order-tracking/
Disallow /returns/
Disallow /refunds/
Disallow /ajax/
Disallow /api/
Disallow /cgi-bin/
Disallow /private/
Disallow /tmp/
Disallow /temp/
Disallow /backup/
Disallow /error/
Disallow /404/
Disallow /500/
Disallow /thank-you/
Disallow /order-received/
Disallow /order-confirmation/
Disallow /payment-success/
Disallow /payment-failed/
Disallow /search
Disallow /search/
Disallow /*?q=*
Disallow /*?search=*
Disallow /*?s=*
Disallow /*?query=*
Disallow /*?keyword=*
Disallow /*?sort=*
Disallow /*?order=*
Disallow /*?orderby=*
Disallow /*?filter=*
Disallow /*?price=*
Disallow /*?color=*
Disallow /*?size=*
Disallow /*?brand=*
Disallow /*?page=*
Disallow /*?p=*
Disallow /*?paged=*
Disallow /*%26page%3D*
Disallow /*%26p%3D*
Disallow /*?view=*
Disallow /*?display=*
Disallow /*?layout=*
Disallow /*?utm_*
Disallow /*%26utm_*
Disallow /*?fbclid=*
Disallow /*?gclid=*
Disallow /*?ref=*
Disallow /*?source=*
Disallow /*?campaign=*
Disallow /*?session=*
Disallow /*?sid=*
Disallow /*?cache=*
Disallow /*?v=*
Disallow /*?ver=*
Disallow /*?version=*
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.docx$
Disallow /*.xls$
Disallow /*.xlsx$
Disallow /*.ppt$
Disallow /*.pptx$
Disallow /*.zip$
Disallow /*.rar$
Disallow /*.tar$
Disallow /*.gz$
Disallow /*.sql$
Disallow /*.log$
Disallow /*.bak$
Allow /css/
Allow /js/
Allow /javascript/
Allow /images/
Allow /img/
Allow /media/
Allow /fonts/
Allow /static/
Allow /assets/
Allow /uploads/
Allow /wp-content/themes/
Allow /wp-content/plugins/
Allow /wp-includes/
Allow /products/
Allow /product/
Allow /categories/
Allow /category/
Allow /collections/
Allow /collections/*
Allow /brands/
Allow /brand/
Allow /sale/
Allow /deals/
Allow /offers/
Allow /clearance/
Allow /new-arrivals/
Allow /blog/
Allow /news/
Allow /about/
Allow /contact/
Allow /shipping/
Allow /delivery/
Allow /size-guide/
Allow /faq/
Allow /privacy-policy/
Allow /terms-conditions/
Allow /cookie-policy/

Other Records

Field Value
crawl-delay 1

googlebot
googlebot-image
googlebot-news

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

pinterest

Rule Path
Disallow /

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mozbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://fbargainsgalore.co.uk/sitemap.xml
sitemap https://fbargainsgalore.co.uk/sitemap_index.xml
sitemap https://fbargainsgalore.co.uk/product-sitemap.xml
sitemap https://fbargainsgalore.co.uk/category-sitemap.xml
sitemap https://fbargainsgalore.co.uk/post-sitemap.xml
sitemap https://fbargainsgalore.co.uk/page-sitemap.xml
Back to top

Comments

  • robots.txt for fbargainsgalore.co.uk
  • Last updated: 2025-09-07
  • Default rules for all crawlers
  • Crawl delay to prevent server overload (1 second)
  • === SECURITY & PRIVACY BLOCKS ===
  • === CONVERSION & THANK YOU PAGES ===
  • === SEARCH & FILTER PARAMETERS ===
  • === DUPLICATE CONTENT PREVENTION ===
  • Sorting and filtering parameters
  • Pagination parameters
  • View parameters
  • Tracking parameters
  • Session and cache parameters
  • === FILE TYPE RESTRICTIONS ===
  • === EXPLICITLY ALLOW IMPORTANT CONTENT ===
  • Essential assets
  • Important pages and sections
  • === SPECIFIC BOT CONFIGURATIONS ===
  • Google bots - more permissive
  • Bing bot
  • Facebook bot (for social sharing)
  • Twitter bot (for social sharing)
  • Pinterest bot
  • SEO tool bots - allow but with slower crawling
  • Block genuinely aggressive or low-value bots
  • === SITEMAP DECLARATIONS ===
Back to top