Log In Sign Up

shephard.co.uk
robots.txt

Robots Exclusion Standard data for shephard.co.uk

Resource Scan

Scan Details

Site Domain shephard.co.uk
Base Domain shephard.co.uk
Scan Status Ok
Last Scan2025-11-03T17:02:46+00:00
Next Scan 2025-12-03T17:02:46+00:00

Last Scan

Scanned2025-11-03T17:02:46+00:00
URL http://shephard.co.uk/robots.txt
Redirect https://shephardmedia.com/robots.txt
Redirect Domain shephardmedia.com
Redirect Base shephardmedia.com
Domain IPs 16.15.186.202, 16.15.194.58, 16.182.96.253, 52.216.43.229, 52.216.93.170, 52.217.173.181, 52.217.73.131, 52.217.90.27
Redirect IPs 104.21.44.142, 172.67.200.196, 2606:4700:3033::6815:2c8e, 2606:4700:3035::ac43:c8c4
Response IP 172.67.200.196
Found Yes
Hash b36b07396e390b2566e9057fc9d8dbb41b9e1537ee2d96eaa2e6c2be9597f730
SimHash 62340d494fa4

Groups

*

Rule Path
Disallow /news/login
Disallow /news/feed/

Other Records

Field Value
crawl-delay 1

petalbot

Rule Path
Disallow /

googleother

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

iboubot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

rss2tg bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

thinkbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.shephardmedia.com/sitemap.xml
Back to top

Comments

  • All use of Shephard Group content is subject to the Terms & Conditions and
  • Copyright Policy set out on shephardmedia.com
Back to top