Log In Sign Up

ogormans.co.uk
robots.txt

Robots Exclusion Standard data for ogormans.co.uk

Resource Scan

Scan Details

Site Domain ogormans.co.uk
Base Domain ogormans.co.uk
Scan Status Ok
Last Scan2026-02-15T00:25:41+00:00
Next Scan 2026-03-17T00:25:41+00:00

Last Scan

Scanned2026-02-15T00:25:41+00:00
URL https://ogormans.co.uk/robots.txt
Domain IPs 104.21.78.184, 172.67.168.85, 2606:4700:3030::6815:4eb8, 2606:4700:3035::ac43:a855
Response IP 172.67.168.85
Found Yes
Hash a18ae1e2427ff7f717179f2a02531ab09200d3b83daa069ca7005860650e246b
SimHash 18386a516793

Groups

googlebot

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

googlebot-product

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-inspectiontool

Rule Path
Disallow

google*

Rule Path
Disallow

bingbot

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

*

Rule Path
Disallow /admin/
Disallow /errors/
Disallow /pkginfo/
Disallow /var/
Disallow /vendor/
Disallow /setup/
Disallow /dev/
Disallow /update/
Disallow /cron.php
Disallow /cron.sh
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.md

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Main crawler
  • Mobile-first crawler (primary)
  • Product & Merchant Center crawlers
  • Image crawler (required for product images)
  • Google Ads landing-page checker
  • Search Console inspection tools
  • Ensure ALL Google bots remain allowed
  • ==========================================
  • ALLOW BING (optional, harmless)
  • ==========================================
  • ==========================================
  • BLOCK AGGRESSIVE SEO CRAWLERS
  • ==========================================
  • ==========================================
  • GENERIC RULES (Safe for Magento)
  • ==========================================
  • Slow down unknown bots
  • Block internal system folders Magento doesn't want indexed
Back to top