Log In Sign Up

crosstrainerhome.co.uk
robots.txt

Robots Exclusion Standard data for crosstrainerhome.co.uk

Resource Scan

Scan Details

Site Domain crosstrainerhome.co.uk
Base Domain crosstrainerhome.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-07-07T06:50:37+00:00
Next Scan 2024-10-05T06:50:37+00:00

Last Successful Scan

Scanned2024-03-10T04:13:09+00:00
URL https://crosstrainerhome.co.uk/robots.txt
Domain IPs 194.1.147.17, 194.1.147.24
Response IP 194.1.147.24
Found Yes
Hash d5970fc4987b0391b2f4dd2ec1f6c90b78a6e8c09b3cc76a2a3195e2641d3960
SimHash 765c59410512

Groups

googlebot

Rule Path
Allow *.js
Allow *.css
Allow /wp-admin/admin-ajax.php
Disallow /?s=
Disallow /search/
Disallow /wp-admin
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /trackback/

*

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin
Disallow /wp-login.php
Disallow /trackback/
Disallow /wp-register.php
Back to top

Comments

  • Allow files critical for rendering
  • Allow AJAX - Do Not Remove
  • Prevent crawl-budget waste on search pages
  • Prevent private admin areas from being crawled
  • Prevent duplicate /feed/ pages from being crawled
  • Disallow: /*/feed/
  • Prevent login page crawls etc
  • Prevent register page crawls etc
  • Prevent Trackback Neg SEO
  • Allow AJAX - Do Not Remove
  • Add all sitemaps
Back to top

Warnings

  • 2 invalid lines.
  • `https` is not a known field.
Back to top