warrencountyrecord.com
robots.txt

Robots Exclusion Standard data for warrencountyrecord.com

Resource Scan

Scan Details

Site Domain warrencountyrecord.com
Base Domain warrencountyrecord.com
Scan Status Ok
Last Scan2024-11-15T23:26:03+00:00
Next Scan 2024-11-22T23:26:03+00:00

Last Scan

Scanned2024-11-15T23:26:03+00:00
URL https://warrencountyrecord.com/robots.txt
Redirect https://www.warrencountyrecord.com/robots.txt
Redirect Domain www.warrencountyrecord.com
Redirect Base warrencountyrecord.com
Domain IPs 65.61.154.4
Redirect IPs 65.61.154.4
Response IP 65.61.154.4
Found Yes
Hash 2d24b3f4370f54bf6425b25844c6073f74d48cfecfbaa976eb83c2bad9bb9291
SimHash 6862fae54fdf

Groups

mediapartners-google

Rule Path
Disallow

magnetbot

Rule Path
Disallow

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

genieo

Rule Path
Disallow /

ecoresearch

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

*

Rule Path
Disallow /css/
Disallow /css_system/
Disallow /js/
Disallow /js_system/
Disallow /ajax/
Disallow /searchmode
Disallow /taxonomy
Disallow /marketplace
Disallow /classifieds
Disallow /premium/
Disallow /premium/agriculture-news
Disallow /premium/automotive-news
Disallow /premium/books-news
Disallow /premium/business-news
Disallow /premium/education-careers
Disallow /premium/entertainment-news
Disallow /premium/food-news
Disallow /premium/garden-news
Disallow /premium/green-living
Disallow /premium/home-news
Disallow /premium/kids-family
Disallow /premium/lifestyle-news
Disallow /premium/money-matters
Disallow /premium/outdoors-news
Disallow /premium/pets-news
Disallow /premium/puzzles
Disallow /premium/real-estate-news
Disallow /premium/seniors-news
Disallow /premium/spanish-news
Disallow /premium/tech-news
Disallow /premium/travel-news
Disallow /premium/games-news
Disallow /premium/health-news
Disallow /calendar

Other Records

Field Value
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-googlenews-victory-1.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-default-victory-1.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-googlenews-westplex-1.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-default-westplex-1.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-googlenews-warren-2.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-googlenews-warren-1.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-default-warren-2.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-default-warren-1.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-googlenews-all-2.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-googlenews-all-1.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-default-all-2.xml
sitemap https://www.warrencountyrecord.com/sitemaps/sitemaps-default-all-1.xml

Comments

  • Global allow for Mediapartners - this is used by Google to place ads in content,
  • not indexing purposes.
  • Global allow for the Klangoo bot
  • Temporary throttles to decrease load during launch
  • Temporary blocks of uninteresting bots
  • Block indexing of search results page
  • Disallow: /site