fitnessfirst.co.uk
robots.txt

Robots Exclusion Standard data for fitnessfirst.co.uk

Resource Scan

Scan Details

Site Domain fitnessfirst.co.uk
Base Domain fitnessfirst.co.uk
Scan Status Ok
Last Scan2024-09-20T06:30:48+00:00
Next Scan 2024-10-04T06:30:48+00:00

Last Scan

Scanned2024-09-20T06:30:48+00:00
URL https://fitnessfirst.co.uk/robots.txt
Redirect https://www.fitnessfirst.co.uk/robots.txt
Redirect Domain www.fitnessfirst.co.uk
Redirect Base fitnessfirst.co.uk
Domain IPs 20.90.134.14
Redirect IPs 20.90.134.14
Response IP 20.90.134.14
Found Yes
Hash e44d361bb4368260874d46da46704cc5f54c9bf70d827f143b4da62fe7e45d5c
SimHash bd293f74b6f2

Groups

*

Rule Path
Disallow /umbraco/
Disallow */?tag=
Disallow */?Tag=
Disallow */?challenge=
Disallow */?club=
Disallow */?category=
Disallow */?Category=
Disallow */?utm_source=
Disallow */?src=
Disallow */?gym=
Disallow /search*
Disallow /login/
Disallow /login/forgot-password
Disallow *class%3D*
Disallow */demandware.static/*

Other Records

Field Value
sitemap https://www.fitnessfirst.co.uk/xml-sitemap

Comments

  • Do not crawl parameter based URLs
  • Do not crawl user account pages
  • Do not crawl old DW/Demandware assets