Log In Sign Up

itsupport.co.uk
robots.txt

Robots Exclusion Standard data for itsupport.co.uk

Resource Scan

Scan Details

Site Domain itsupport.co.uk
Base Domain itsupport.co.uk
Scan Status Ok
Last Scan2025-11-11T00:09:28+00:00
Next Scan 2025-12-11T00:09:28+00:00

Last Scan

Scanned2025-11-11T00:09:28+00:00
URL https://itsupport.co.uk/robots.txt
Domain IPs 194.0.252.52
Response IP 194.0.252.52
Found Yes
Hash 312a18c7c3c0950f2948f50d46e9d4d0ede194f0825b1b706eb601947b865180
SimHash 2d7edc5366c0

Groups

*

Rule Path
Disallow /wp/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /xmlrpc.php
Disallow /cms-login
Disallow /app/plugins/
Disallow /app/mu-plugins/
Disallow /app/themes/
Disallow /composer.json
Disallow /composer.lock
Disallow /vendor/
Disallow /config/
Disallow /.env
Disallow /readme.html
Disallow /license.txt
Disallow /cgi-bin/
Disallow /*?*

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.select-technology.co.uk/sitemap.xml
Back to top

Comments

  • Allow all good bots, then selectively disallow specific crawlers
  • Don't crawl WP core
  • Don't crawl Bedrock app folders
  • Don't crawl build/config/vendor files
  • Block misc. 'support' files that don't need to be indexed
  • Block query-string URLs
  • Optional: crawl-delay for polite bots
  • Block known heavy SEO/spam crawlers
  • Declare sitemap at the end
Back to top