Log In Sign Up

gibbs-gillespie.co.uk
robots.txt

Robots Exclusion Standard data for gibbs-gillespie.co.uk

Resource Scan

Scan Details

Site Domain gibbs-gillespie.co.uk
Base Domain gibbs-gillespie.co.uk
Scan Status Ok
Last Scan2025-12-06T18:21:09+00:00
Next Scan 2026-01-05T18:21:09+00:00

Last Scan

Scanned2025-12-06T18:21:09+00:00
URL https://gibbs-gillespie.co.uk/robots.txt
Redirect https://www.gibbs-gillespie.co.uk/robots.txt
Redirect Domain www.gibbs-gillespie.co.uk
Redirect Base gibbs-gillespie.co.uk
Domain IPs 159.89.250.66
Redirect IPs 159.89.250.66
Response IP 159.89.250.66
Found Yes
Hash e88c65ac3d7ba42f733b51b61309be88bd421626c3b897be433ff6c8ad59b94e
SimHash c9161ff4eb19

Groups

*

Rule Path
Allow /
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/*.php
Disallow /email/
Disallow /functions/
Disallow /includes/
Disallow /installation/
Disallow /json/
Disallow /language/
Disallow /libraries/
Disallow /log/
Disallow /logs/
Disallow /modules/*.php
Disallow /plugins/
Disallow /server/
Disallow /status.txt
Disallow /tmp/
Disallow /api/
Allow /cache/stb/assets/

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

blexbot

Rule Path
Disallow /

spriftcrawler

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

cbbot

Rule Path
Disallow /

oodlebot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

verity/1.1

Rule Path
Disallow /

proximic

Rule Path
Disallow /

petalbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.gibbs-gillespie.co.uk/sitemap.xml
Back to top

Comments

  • Mozilla/5.0 (compatible; proximic; +www.proximic.com/info/spider.php)
  • blocked as they are making incorrect requests
  • Petalbot blocked as they are making a huge number of requests
Back to top

Warnings

  • 2 invalid lines.
  • `host` is not a known field.
Back to top