Log In Sign Up

bakeryandsnacks.com
robots.txt

Robots Exclusion Standard data for bakeryandsnacks.com

Resource Scan

Scan Details

Site Domain bakeryandsnacks.com
Base Domain bakeryandsnacks.com
Scan Status Ok
Last Scan2025-03-06T01:34:32+00:00
Next Scan 2025-04-05T01:34:32+00:00

Last Scan

Scanned2025-03-06T01:34:32+00:00
URL https://bakeryandsnacks.com/robots.txt
Redirect https://www.bakeryandsnacks.com:443/robots.txt
Redirect Domain www.bakeryandsnacks.com
Redirect Base bakeryandsnacks.com
Domain IPs 15.197.68.198, 15.197.82.42
Redirect IPs 173.222.148.43, 23.49.60.58, 2600:1413:b000:13::b857:c184, 2600:1413:b000:13::b857:c193
Response IP 173.222.148.43
Found Yes
Hash 918f55cf4e61c40e754b8688518af057703199792dfc4ed6ab038909c98e168b
SimHash 741ef310e49d

Groups

*

Rule Path
Disallow /search/
Disallow /newsletter/
Disallow */pf/api/

mozilla/5.0 (x11; linux x86_64; rv:87.0) gecko/20100101 firefox/87.0;onetrustbot;

Rule Path
Disallow

anthropic-ai
bytespider
chatgpt
claudebot
gptbot
oai-searchbot

Rule Path
Disallow /search/
Disallow /newsletter/
Disallow */pf/api/

Other Records

Field Value
crawl-delay 1

ahrefsbot
ahrefssiteaudit
amazonbot
archive-it
audacy-podcast-scraper
discordbot
facebookexternalhit
feedburner
feedly
feedparser
feedvalidator
flipboard
gofeed
hubspot crawler
netnewswire
pinterestbot
twitterbot
whatsapp

Rule Path
Disallow /search/
Disallow /newsletter/
Disallow */pf/api/

Other Records

Field Value
crawl-delay 1

adsbot
aliyunsecbot
archivebot
awariobot
backlinksextendedbot
baiduspider
barkrowler
birdcrawlerbot
bitsightbot
blackboardally
blexbot
blogtrottr
bluechipbacklinks
bpimagewalker
bravebot
ccbot
cincraw
cis5550-crawler
cloudservermarketspider
coccocbot
coibotparser
crsspxlbot
cyberfindcrawler
cyotekwebcopy
darwin
dataforseobot
deepcrawl
diffbot
domainsbot
domainstatsbot
domcopbot
dotbot
dragonbot
dubbotbot
emailwolf
equellaurlbot
ev-crawler
exabot
facebot
faraday
feedbot
filemaker
glmslinkanalysis
grapeshotcrawler
harsilbot
iaskspider
imagesiftbot
inetdex-bot
informa java api
intently
io_bot
isscyberriskcrawler
istellabot
kazbtbot
linguee bot
linkanalyser
linkdexbot
linkfluence
livelapbot
magus bot
maxthon
mixrankbot
mj12bot
mj12bot
mojeekbot
monsidobot
moodlebot
netcraftsurveyagent
netestate ne crawler
neticle crawler
netpeak
newsify
nicecrawler
nodeping
online-webceo-bot
openindexspider
orbbot
paqlebot
petalbot
pocketparser
pubmatic crawler bot
rankurbot
scrapy
searchbot
sebot
seekport crawler
seekportbot
seekrbot
semanticscholarbot
sentibot
seo-audit-check-bot
seobilitybot
serpstatbot
seznambot
seznambot
simplepie
siteauditbot
sitebulb
siteone-crawler
smtbot
sprinklr
srcedamp
staffbase
startmebot
storebot
superbot
superpagesbot2
surdotlybot
tentacles
timpibot
trendictionbot0
turnitin
uk_lddc_bot
ultimate_sitemap_parser
vebidoobot
velenpublicwebcrawler
wallabyupbot
webwikibot
wellknownbot
wepchsearchengine
xbot
yeti
yisouspider
zoominfobot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/news-sitemap/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-section/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/videos/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/promotional-features/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/product-innovations/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/suppliers/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/events/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Business/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Manufacturers/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Industry-voices/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Retailer-shopper-insights/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Ingredients/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Science/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Processing-packaging/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Regulation-policy-food-safety/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Commodities-milling-grains/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/News/Reformulation/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Sectors/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Sectors/Bread-dough-products/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Sectors/Cakes-pastries/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Sectors/Biscuits/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Sectors/Snacks/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Sectors/Breakfast-cereals/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Sectors/Desserts-frozen-goods/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/Allergens/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/Health/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/Ancient-grains/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/Sustainability/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/Artisan-craft-market/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/Clean-label/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/Free-from/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Trends/HFSS/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Campbell-Snacks/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/General-Mills/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Grupo-Bimbo/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Kellanova-WK-Kellogg-Co/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/KIND/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/KP-Snacks/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Kraft-Foods/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Mondelez/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Nestle/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/PepsiCo-Frito-Lay/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Pladis/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Post-Holdings/
sitemap https://www.bakeryandsnacks.com/arc/outboundfeeds/sitemap-index/category/Big-brands/Start-ups-disruptors/
Back to top

Comments

  • Block all robots going to the following directories
  • OneTrustBot can go anywhere
  • AI bots specific crawl-delay
  • Other bots with specific crawl-delay
  • Unwanted bots
  • Sitemaps
Back to top

Warnings

  • 3 invalid lines.
Back to top