Log In Sign Up

agriculturedive.com
robots.txt

Robots Exclusion Standard data for agriculturedive.com

Resource Scan

Scan Details

Site Domain agriculturedive.com
Base Domain agriculturedive.com
Scan Status Ok
Last Scan2026-01-04T13:39:11+00:00
Next Scan 2026-01-11T13:39:11+00:00

Last Scan

Scanned2026-01-04T13:39:11+00:00
URL https://agriculturedive.com/robots.txt
Redirect https://www.agriculturedive.com/robots.txt
Redirect Domain www.agriculturedive.com
Redirect Base agriculturedive.com
Domain IPs 104.18.41.133, 172.64.146.123, 2606:4700:440d::ac40:927b, 2a06:98c1:3103::6812:2985
Redirect IPs 104.18.41.133, 172.64.146.123, 2606:4700:440d::ac40:927b, 2a06:98c1:3103::6812:2985
Response IP 104.18.41.133
Found Yes
Hash 00fb88aa4b686c39176c3fd66d03c69e22d976ff1a34926199cc2b7403ea32a0
SimHash fb7683114550

Groups

*

Rule Path
Disallow /admin/
Disallow /newsletter/
Disallow /healthcheck/
Disallow /subpage/
Disallow /ckeditor/
Disallow /api/
Disallow /static/images/
Disallow /subscriber/
Disallow /selfservice/
Disallow /press-release/preview/
Disallow /press-release/editor/
Disallow /search/*
Allow /search/$
Disallow /user_media/
Disallow /imgproxy/
Disallow /topic/?page=*
Allow /sitemap.xml

Other Records

Field Value
crawl-delay 5

twitterbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

googlebot-news

Rule Path
Disallow /admin/
Disallow /newsletter/
Disallow /healthcheck/
Disallow /subpage/
Disallow /ckeditor/
Disallow /api/
Disallow /static/images/
Disallow /subscriber/
Disallow /selfservice/
Disallow /press-release/preview/
Disallow /press-release/editor/
Disallow /search/*
Allow /search/$
Allow /google_news_sitemap.xml
Allow /user_media/
Allow /imgproxy/
Allow /static/images/favicons

googlebot-image

Rule Path
Disallow /
Allow /imgproxy/
Allow /static/images/favicons
Allow /favicon.ico
Allow /white-favicon.ico
Allow /apple-touch-icon.png
Allow /favicon-32x32.png
Allow /white-favicon-32x32.png
Allow /favicon-16x16.png
Allow /white-favicon-16x16.png
Allow /site.webmanifest
Allow /safari-pinned-tab.svg
Allow /browserconfig.xml
Allow /android-chrome-144x144.png
Allow /mstile-150x150.png

petalbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

claritybot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /admin/
Disallow /newsletter/
Disallow /healthcheck/
Disallow /subpage/
Disallow /ckeditor/
Disallow /api/
Disallow /static/images/
Disallow /subscriber/
Disallow /selfservice/
Disallow /press-release/preview/
Disallow /press-release/editor/
Disallow /search/*
Allow /search/$
Disallow /user_media/
Disallow /imgproxy/
Disallow /topic/
Disallow /editors/

Other Records

Field Value
crawl-delay 30

amazonbot

Rule Path
Disallow /admin/
Disallow /newsletter/
Disallow /healthcheck/
Disallow /subpage/
Disallow /ckeditor/
Disallow /api/
Disallow /static/images/
Disallow /subscriber/
Disallow /selfservice/
Disallow /press-release/preview/
Disallow /press-release/editor/
Disallow /search/*
Allow /search/$
Disallow /user_media/
Disallow /imgproxy/
Disallow /topic/
Disallow /editors/
Disallow /signup/*
Allow /signup/$

Other Records

Field Value
crawl-delay 10

amazon-kendra

Product Comment
amazon-kendra Amazon Kendra Web Crawler
Rule Path
Disallow /
Back to top

Comments

  • ..;coxkOOOOOOkxoc;'.
  • .:d0NWMMMMMMMMMMMMMMWN0xc'
  • .:kXMMMMMMMMMMMMMMMMMMMMMMMXl.
  • .c0WMMMMMMMMMMMMMMMMMMMMMMMXd'
  • ,OWMMMMMMMMMMMMMMMMMMMMMMMXo' ..
  • cXMMMMMMXo::::::::::::::col. .lKXl.
  • lNMMMMMMM0' .lKWMMNo
  • :XMMMMMMMM0' .l0WMMMMMNc
  • .OMMMMMMMMM0' .ccccccc;. ,KMMMMMMMMO.
  • :NMMMMMMMMM0' oWMMMMMMWKc. oWMMMMMMMN:
  • lWMMMMMMMMM0' oWMMMMMMMMX: ,KMMMMMMMMo
  • oMMMMMMMMMM0' oWMMMMMMMMNc ,KMMMMMMMMd
  • cNMMMMMMMMM0' oWMMMMMMMNd. lWMMMMMMMWl
  • '0MMMMMMMMWk. ,oooooooc' ,0MMMMMMMMK,
  • oWMMMMMMXo. ,0MMMMMMMMWo
  • .xWMMMXd' ,dXMMMMMMMMWk.
  • .xWNx' .',''''''',,;coONMMMMMMMMMWk.
  • .:, .l0WWWWWWWWWWWMMMMMMMMMMMMMNd.
  • .lKWMMMMMMMMMMMMMMMMMMMMMMMWk;
  • .lKWMMMMMMMMMMMMMMMMMMMMMMMNk;.
  • .ckXWMMMMMMMMMMMMMMMMMMWXkl'
  • .;ldO0XNWWWWWWNXKOxl;.
  • ..'',,,,''..
  • NOTE: Allow is a non-standard directive for robots.txt. It is allowed by Google bots. See https://developers.google.com/search/reference/robots_txt#allow
  • Crawl delay asks bots to wait this many seconds between requests. Ignored by google.
  • no deep queries to search
  • don't index our dynamic images
  • don't deep index topic pages
  • Rules for specific crawlers below. Note that these don't stack. If you create a specific user-agent rule
  • you should copy the rules over from '*' above by hand.
  • Allow Twitter to see all links
  • Allow Googlebot-News to see header images and favicons, BUT make it follow all the directives from our * group
  • See below link for why we have to repeat these directives
  • https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-user-agents
  • no deep queries to search
  • Allow Google News to see header images and favicons
  • Googlebot-Image is now used for favicons. Allow it to see favicon-related files but nothing else
  • Don't let PetalBot crawl at all
  • Block sentione.com
  • block seoclarity.net/bot.html
  • block omgili.com/crawler.html
  • All Facebook crawler user-agent to see all
  • We want this bot to crawl way slower https://ahrefs.com/robot/
  • no deep queries to search
  • don't index our dynamic images
  • Restrict what Amazonbot (Alexa) can see, and a
  • no deep queries to search
  • don't index our dynamic images
Back to top