Log In Sign Up

defenceshare.mod.uk
robots.txt

Robots Exclusion Standard data for defenceshare.mod.uk

Resource Scan

Scan Details

Site Domain defenceshare.mod.uk
Base Domain mod.uk
Scan Status Ok
Last Scan2025-08-17T06:19:36+00:00
Next Scan 2025-09-16T06:19:36+00:00

Last Scan

Scanned2025-08-17T06:19:36+00:00
URL https://defenceshare.mod.uk/robots.txt
Domain IPs 31.25.189.123
Response IP 31.25.189.123
Found Yes
Hash c87c11a510e85ab9b7d96b84c9ff84c95a07caceb33ed0f779ca1533ad6adfb2
SimHash 525471c1d5f4

Groups

*

Rule Path
Disallow /

baiduspider
twiceler
jikespider
ezooms
ahrefsbot
spbot
mj12bot
domain re-animator bot
slurp
yandex
sputnikbot
applebot
megaindex.ru/2.0
mauibot
semrushbot
sogou web spider
screaming frog seo spider
seekportbot
petalbot
amazonbot
bytespider
gptbot
google-extended

Rule Path
Disallow /

googlebot
bingbot

Rule Path
Allow /
Disallow /kz/
Disallow /connect.ti/
Disallow /consult.ti/
Disallow /inovem.ti/
Disallow /system/register
Disallow /system/login
Disallow /system/text
Disallow /system/forgotPassword
Disallow /system/mailToSiteOwner
Disallow /system/contactSiteOwner

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Make it extra clear for specific crawlers
  • Allow google and bing
  • Old style urls
  • New urls
  • Maximum rate is one page every 10 seconds
  • Only visit between certain hours (UTC)
Back to top

Warnings

  • `visit-time` is not a known field.
Back to top