Log In Sign Up

q-hub.co.uk
robots.txt

Robots Exclusion Standard data for q-hub.co.uk

Resource Scan

Scan Details

Site Domain q-hub.co.uk
Base Domain q-hub.co.uk
Scan Status Ok
Last Scan2025-10-23T07:01:37+00:00
Next Scan 2025-11-22T07:01:37+00:00

Last Scan

Scanned2025-10-23T07:01:37+00:00
URL https://q-hub.co.uk/robots.txt
Redirect https://www.q-hub.app/robots.txt
Redirect Domain www.q-hub.app
Redirect Base q-hub.app
Domain IPs 35.214.4.54
Redirect IPs 13.203.125.58, 13.233.175.166, 3.109.243.18
Response IP 52.68.134.190
Found Yes
Hash c43eddbcb388c71368ce031b4d7a4adaf8139ac8a7f8d068f1b9f248884cd2f4
SimHash 29575d737463

Groups

*

Rule Path
Disallow /*?*
Disallow /preview/
Disallow /draft/
Disallow /admin/
Disallow /login/
Disallow /cart/
Disallow /checkout/
Disallow /search/
Disallow /404/
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.xls$
Disallow /*.zip$
Allow /css/
Allow /js/
Allow /images/

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

seobilitybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

seoprofiler

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

woorank

Rule Path
Disallow /

siteanalyzerbot

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

ranksonicsiteauditor

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

linkexaminer

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Disallow /search/
Disallow /*?*

bingbot

Rule Path
Allow /
Disallow /search/
Disallow /*?*

googlebot

Rule Path
Allow /rendered-content/
Disallow /render-blocking/
Disallow /wp-content/debug.log
Disallow /error_log
Disallow /readme.html
Disallow /.git/
Disallow /.env

duckduckbot

Rule Path
Disallow /large-images/
Disallow /assets/
Allow /*.css$
Allow /*.js$
Allow /*.jpg$
Allow /*.png$
Back to top

Other Records

Field Value
sitemap https://www.q-hub.app/sitemap.xml
Back to top

Comments

  • robots.txt for Q-Hub
  • Last updated: 2025-02-06
  • Notes:
  • - Advanced directives for precision crawl management
  • - Combines query handling, resource prioritisation, and aggressive bot control
  • General Rules for All Bots
  • Block query parameters to prevent duplicate indexing
  • Disallow internal/private sections
  • Block unnecessary file types to save crawl budget
  • Allow necessary resources
  • Specific Bot Rules - Block known aggressive or unnecessary bots
  • Google-Specific Rules
  • Bing-Specific Rules
  • Query Parameter Optimisation
  • Advanced Techniques
  • Prioritise JavaScript-rendered content
  • Block render-blocking files if applicable
  • Prevent discovery of sensitive files
  • Bots that should crawl but avoid heavy resources
  • Advanced Caching Directives - Allow static assets for efficient caching
  • Comments for Future Updates:
  • - Regularly monitor server logs for unusual bot behaviour.
  • - Update the blocked bots list based on crawler trends.
  • Sitemap
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top