mexicoledger.com
robots.txt

Robots Exclusion Standard data for mexicoledger.com

Resource Scan

Scan Details

Site Domain mexicoledger.com
Base Domain mexicoledger.com
Scan Status Ok
Last Scan2024-06-13T12:06:22+00:00
Next Scan 2024-06-20T12:06:22+00:00

Last Scan

Scanned2024-06-13T12:06:22+00:00
URL https://mexicoledger.com/robots.txt
Domain IPs 65.61.154.4
Response IP 65.61.154.4
Found Yes
Hash 291be4b5cdc5d9f6cf1f65649b96f4415310d505f1b5ed0449c7e964862e74e8
SimHash 6862dafd455f

Groups

mediapartners-google

Rule Path
Disallow

magnetbot

Rule Path
Disallow

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

genieo

Rule Path
Disallow /

ecoresearch

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

*

Rule Path
Disallow /css/
Disallow /css_system/
Disallow /js/
Disallow /js_system/
Disallow /ajax/
Disallow /searchmode
Disallow /taxonomy
Disallow /marketplace
Disallow /classifieds
Disallow /premium/
Disallow /premium/agriculture-news
Disallow /premium/automotive-news
Disallow /premium/books-news
Disallow /premium/business-news
Disallow /premium/education-careers
Disallow /premium/entertainment-news
Disallow /premium/food-news
Disallow /premium/garden-news
Disallow /premium/green-living
Disallow /premium/home-news
Disallow /premium/kids-family
Disallow /premium/lifestyle-news
Disallow /premium/money-matters
Disallow /premium/outdoors-news
Disallow /premium/pets-news
Disallow /premium/puzzles
Disallow /premium/real-estate-news
Disallow /premium/seniors-news
Disallow /premium/spanish-news
Disallow /premium/tech-news
Disallow /premium/travel-news
Disallow /premium/games-news
Disallow /premium/health-news
Disallow /calendar

Comments

  • Global allow for Mediapartners - this is used by Google to place ads in content,
  • not indexing purposes.
  • Global allow for the Klangoo bot
  • Temporary throttles to decrease load during launch
  • Temporary blocks of uninteresting bots
  • Block indexing of search results page
  • Disallow: /site