demon-tweeks.co.uk
robots.txt
Robots Exclusion Standard data for demon-tweeks.co.uk
Resource Scan
Scan Details
| Site Domain | demon-tweeks.co.uk |
| Base Domain | demon-tweeks.co.uk |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-11-11T00:49:13+00:00 |
| Next Scan | 2025-12-11T00:49:13+00:00 |
Last Successful Scan
| Scanned | 2025-09-19T12:51:01+00:00 |
| URL | https://demon-tweeks.co.uk/robots.txt |
| Domain IPs | 104.21.75.115, 172.67.222.205, 2606:4700:3031::6815:4b73, 2606:4700:3035::ac43:decd |
| Response IP | 104.21.75.115 |
| Found | Yes |
| Hash | 23ef6a8be6e81f08da400bb26e54bdd927c45af844674ab5e651dc234756ec0d |
| SimHash | 8d6ce9f1ef30 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | */shop-by/ |
| Allow | */shop-by/page/ |
| Allow | */shop-by/categories/*/page/ |
| Allow | /*/*/*/*/shop-by/brand/ |
| Allow | /*/*/*/shop-by/brand/ |
| Disallow | /*/*/*/*/shop-by/brand/*%2C*/ |
| Disallow | /*/*/*/shop-by/brand/*%2C*/ |
| Disallow | /uk/*/*/shop-by/brand/ |
| Disallow | /us/*/*/shop-by/brand/ |
| Disallow | /au/*/*/shop-by/brand/ |
| Disallow | /nz/*/*/shop-by/brand/ |
| Disallow | /jp/*/*/shop-by/brand/ |
| Disallow | /dk/*/*/shop-by/brand/ |
| Disallow | /se/*/*/shop-by/brand/ |
| Disallow | /no/*/*/shop-by/brand/ |
| Disallow | /eu/*/*/shop-by/brand/ |
| Disallow | /404/ |
| Disallow | /app/ |
| Disallow | /cgi-bin/ |
| Disallow | /downloader/ |
| Disallow | /includes/ |
| Disallow | /lib/ |
| Disallow | /pkginfo/ |
| Disallow | /report/ |
| Disallow | /magento/ |
| Disallow | /stats/ |
| Disallow | /var/ |
| Disallow | /index.php/ |
| Disallow | /catalog/product_compare/ |
| Disallow | /catalog/category/view/ |
| Disallow | /catalog/product/view/ |
| Disallow | /catalogsearch/ |
| Disallow | /checkout/ |
| Disallow | /control/ |
| Disallow | /contacts/ |
| Disallow | /customer/ |
| Disallow | /wishlist/ |
| Disallow | /review/ |
| Disallow | /sendfriend/ |
| Disallow | /tag/ |
| Disallow | /search/* |
| Disallow | /sales/ |
| Disallow | /*?p=*& |
| Disallow | /cron.php |
| Disallow | /cron.sh |
| Disallow | /error_log |
| Disallow | /install.php |
| Disallow | /LICENSE.html |
| Disallow | /LICENSE.txt |
| Disallow | /LICENSE_AFL.txt |
| Disallow | /STATUS.txt |
| Disallow | /*?brand=* |
| Disallow | /*?product_list_order=* |
| Disallow | /*?source=* |
| Disallow | /*?width=* |
| Disallow | /*?diameter=* |
| Disallow | /*?size=* |
| Disallow | /*?colour_group=* |
| Disallow | /*?pcd=* |
| Disallow | /*?helmet_type=* |
| Disallow | /*?approval=* |
| Disallow | /*?attachment_points=* |
| Disallow | /*?price=* |
| Disallow | /*?badge=* |
| Disallow | /*?dm_i=* |
| Disallow | /*?oil_viscosity=* |
| Disallow | /*?tyre_profile=* |
| Disallow | /*?gender=* |
| Disallow | /*?material=* |
| Disallow | /*?hans_fhr_posts=* |
| Disallow | /*?cat=* |
| Disallow | /*?gauge_type=* |
| Disallow | /*?a=* |
| Disallow | /*?damping_type=* |
| Disallow | /*?oil_type=* |
| Disallow | /*?dish=* |
| Disallow | /*?application=* |
| Disallow | /*?thickness=* |
| Disallow | /*?communication=* |
| Disallow | /*?limit=* |
| Disallow | /*?fitment=* |
| Disallow | /*?car_seat_age_range=* |
| Disallow | /*?pressure=* |
| Disallow | /*?battery_type=* |
| Disallow | /*?thread=* |
| Disallow | /*?pedal_option=* |
| Disallow | /*?operation=* |
| Disallow | /*?bulb_type=* |
| Disallow | /*?helmet_design=* |
| Disallow | /*?silicone_hose_type=* |
| Disallow | /*?wbraid=* |
| Disallow | /*?tyre_speed_rating=* |
| Disallow | /*?fuel_additive_type=* |
| Disallow | /*?mount_type=* |
| Disallow | /*?battery_capacity=* |
| Disallow | /*?discipline=* |
| Disallow | /*?display=* |
| Disallow | /*?caliper_number_of_pistons=* |
| Disallow | /*?run_flat=* |
| Disallow | /*?angle=* |
| Disallow | /*?buckle_type=* |
| Disallow | /*?fhr_recline=* |
| Disallow | /*?hose_end_fitting=* |
| Disallow | /*?length=* |
| Disallow | /*?pad_compound=* |
| Disallow | /*?adaptor_type=* |
| Disallow | /*?facing=* |
| Disallow | /*?brake_component=* |
| Disallow | /*?totalCount=* |
| Disallow | /*?ref=* |
| Disallow | /*?caliper_model=* |
| Disallow | /*?attachment=* |
| Disallow | /*?neck_type=* |
| Disallow | /*?fhr_compatibility=* |
| Disallow | /*?cooler_height=* |
| Disallow | /*utm_source%3D* |
| Disallow | /*?q=* |
| Disallow | /*?awc=* |
| Disallow | /*?hose_size=* |
| Disallow | /*?thread_size=* |
| Disallow | /*?fluid_specification=* |
| Disallow | /*?rev_range=* |
| Disallow | */customer/account/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_uk.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_us.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_au.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_nz.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_jp.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_dk.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_se.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_no.xml |
| sitemap | https://www.demon-tweeks.com/media/sitemaps/sitemap_eu.xml |
Comments