Log In Sign Up

morningadvertiser.co.uk
robots.txt

Robots Exclusion Standard data for morningadvertiser.co.uk

Resource Scan

Scan Details

Site Domain morningadvertiser.co.uk
Base Domain morningadvertiser.co.uk
Scan Status Ok
Last Scan2025-03-04T07:37:41+00:00
Next Scan 2025-04-03T07:37:41+00:00

Last Scan

Scanned2025-03-04T07:37:41+00:00
URL https://morningadvertiser.co.uk/robots.txt
Redirect https://www.morningadvertiser.co.uk:443/robots.txt
Redirect Domain www.morningadvertiser.co.uk
Redirect Base morningadvertiser.co.uk
Domain IPs 15.197.68.198, 15.197.82.42
Redirect IPs 23.52.171.122, 23.52.171.137, 2600:1413:b000:13::b857:c184, 2600:1413:b000:13::b857:c193
Response IP 23.52.171.137
Found Yes
Hash 22d45ac033048f73e93025d885753459c8dc2794056e4903a71691dbddd9438e
SimHash f41ef118e697

Groups

*

Rule Path
Disallow /search/
Disallow /newsletter/
Disallow */pf/api/

mozilla/5.0 (x11; linux x86_64; rv:87.0) gecko/20100101 firefox/87.0;onetrustbot;

Rule Path
Disallow

anthropic-ai
bytespider
chatgpt
claudebot
gptbot
oai-searchbot

Rule Path
Disallow /search/
Disallow /newsletter/
Disallow */pf/api/

Other Records

Field Value
crawl-delay 1

ahrefsbot
ahrefssiteaudit
amazonbot
archive-it
audacy-podcast-scraper
discordbot
facebookexternalhit
feedburner
feedly
feedparser
feedvalidator
flipboard
gofeed
hubspot crawler
netnewswire
pinterestbot
twitterbot
whatsapp

Rule Path
Disallow /search/
Disallow /newsletter/
Disallow */pf/api/

Other Records

Field Value
crawl-delay 1

adsbot
aliyunsecbot
archivebot
awariobot
backlinksextendedbot
baiduspider
barkrowler
birdcrawlerbot
bitsightbot
blackboardally
blexbot
blogtrottr
bluechipbacklinks
bpimagewalker
bravebot
ccbot
cincraw
cis5550-crawler
cloudservermarketspider
coccocbot
coibotparser
crsspxlbot
cyberfindcrawler
cyotekwebcopy
darwin
dataforseobot
deepcrawl
diffbot
domainsbot
domainstatsbot
domcopbot
dotbot
dragonbot
dubbotbot
emailwolf
equellaurlbot
ev-crawler
exabot
facebot
faraday
feedbot
filemaker
glmslinkanalysis
grapeshotcrawler
harsilbot
iaskspider
imagesiftbot
inetdex-bot
informa java api
intently
io_bot
isscyberriskcrawler
istellabot
kazbtbot
linguee bot
linkanalyser
linkdexbot
linkfluence
livelapbot
magus bot
maxthon
mixrankbot
mj12bot
mj12bot
mojeekbot
monsidobot
moodlebot
netcraftsurveyagent
netestate ne crawler
neticle crawler
netpeak
newsify
nicecrawler
nodeping
online-webceo-bot
openindexspider
orbbot
paqlebot
petalbot
pocketparser
pubmatic crawler bot
rankurbot
scrapy
searchbot
sebot
seekport crawler
seekportbot
seekrbot
semanticscholarbot
sentibot
seo-audit-check-bot
seobilitybot
serpstatbot
seznambot
seznambot
simplepie
siteauditbot
sitebulb
siteone-crawler
smtbot
sprinklr
srcedamp
staffbase
startmebot
storebot
superbot
superpagesbot2
surdotlybot
tentacles
timpibot
trendictionbot0
turnitin
uk_lddc_bot
ultimate_sitemap_parser
vebidoobot
velenpublicwebcrawler
wallabyupbot
webwikibot
wellknownbot
wepchsearchengine
xbot
yeti
yisouspider
zoominfobot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/news-sitemap/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-section/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/videos/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/promotional-features/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/product-innovations/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/suppliers/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/events/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/News/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Trends-Analysis/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Opinion-Interviews/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Investments/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/New-Product-Development/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Events/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Beer/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Cask/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Cider/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Low-No-Alcohol/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Soft-Hot-Drinks/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Spirits-Cocktails/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Wine/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Food-Trends/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Ingredients/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Menu-Ideas/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Food-Drink/Snacks/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Legal/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Legal/Health-Safety/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Legal/Legislation/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Legal/Licensing-Law/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Legal/Property-Law/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Design-Furnishings/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Entertainment-Events/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Environmental-Social-Governance/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Equipment/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Financial/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Marketing/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Staffing/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Operations/Technology/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Admiral-Taverns/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Greene-King/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Marston-s/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Punch-Pubs-Co/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Star-Pubs-Bars/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Stonegate-Group/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Independent-Operators/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Managed-Groups/
sitemap https://www.morningadvertiser.co.uk/arc/outboundfeeds/sitemap-index/category/Companies/Gastropubs/
Back to top

Comments

  • Block all robots going to the following directories
  • OneTrustBot can go anywhere
  • AI bots specific crawl-delay
  • Other bots with specific crawl-delay
  • Unwanted bots
  • Sitemaps
Back to top

Warnings

  • 3 invalid lines.
Back to top