retrobike.co.uk
robots.txt

Robots Exclusion Standard data for retrobike.co.uk

Resource Scan

Scan Details

Site Domain retrobike.co.uk
Base Domain retrobike.co.uk
Scan Status Ok
Last Scan2025-09-30T23:05:57+00:00
Next Scan 2025-10-07T23:05:57+00:00

Last Scan

Scanned2025-09-30T23:05:57+00:00
URL https://retrobike.co.uk/robots.txt
Redirect https://www.retrobike.co.uk/robots.txt
Redirect Domain www.retrobike.co.uk
Redirect Base retrobike.co.uk
Domain IPs 104.21.53.204, 172.67.218.177, 2606:4700:3035::6815:35cc, 2606:4700:3036::ac43:dab1
Redirect IPs 104.21.53.204, 172.67.218.177, 2606:4700:3035::6815:35cc, 2606:4700:3036::ac43:dab1
Response IP 104.21.53.204
Found Yes
Hash d81f3600af47b6aceee24a5d756b5d2fbaf31496547271bcc0ef0563c756b6c5
SimHash 38647d816152

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /forum/admin/
Disallow /forum/cache/
Disallow /forum/db/
Disallow /forum/images/
Disallow /forum/includes/
Disallow /forum/languages/
Disallow /forum/templates/
Disallow /forum/common.php
Disallow /forum/config.php
Disallow /forum/extension.inc
Disallow /forum/faq.php
Disallow /forum/groupcp.php
Disallow /forum/login.php
Disallow /forum/memberlist.php
Disallow /forum/modcp.php
Disallow /forum/posting.php
Disallow /forum/privmsg.php
Disallow /forum/profile.php
Disallow /forum/search.php
Disallow /forum/viewonline.php
Disallow /BMW/
Disallow /css/
Disallow /gshock/
Disallow /store/
Disallow /phpld/
Disallow /auction/results.php
Disallow /auction/results2.php
Disallow /auction/resultsnew.php
Disallow /auction/resultsdesc.php
Disallow /forum/google_map.php?lat
Disallow /wp-content/plugins/
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /openx/www/admin/
Disallow /w/
Disallow /wiki/Special%3ASearch
Disallow /wiki/Special%3ARandom
Disallow /forum/gallery2.php

Comments

  • Throttle bing
  • Misc
  • Wordpress
  • OpenX
  • Wiki rules
  • gallery2 rules
  • Disallow: /gallery2/main.php?g2_view=keyalbum.KeywordAlbum&g2_keyword
  • loads of errors on webmastertools so remove URL hope all spidered