Log In Sign Up

frigorificos365.com
robots.txt

Robots Exclusion Standard data for frigorificos365.com

Resource Scan

Scan Details

Site Domain frigorificos365.com
Base Domain frigorificos365.com
Scan Status Ok
Last Scan2024-10-10T18:31:54+00:00
Next Scan 2024-10-17T18:31:54+00:00

Last Scan

Scanned2024-10-10T18:31:54+00:00
URL https://frigorificos365.com/robots.txt
Domain IPs 104.21.44.23, 172.67.193.246, 2606:4700:3031::6815:2c17, 2606:4700:3035::ac43:c1f6
Response IP 104.21.44.23
Found Yes
Hash 563092d3fa9533c987e4d2127dad8a9e180db389531277eb0209060792485eca
SimHash 7398795bd4fc

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

google-hoteladsverifier

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

berlin-fu-cow

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

semager

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /home/

mj12bot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

deusu

Rule Path
Disallow /

meds-online24.com

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

pingdom.com

Rule Path
Disallow /

prtgcloudbot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bubing

Rule Path
Disallow /

lcc

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mauibot+

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /
Back to top

Comments

  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • COW - Proyecto universitario que escanea webs. De todas maneras no va a obtener mucha informacion de nuestra web
  • http://hpsg.fu-berlin.de/cow/
  • JikeSpider - Crawler general de uso privado. No nos reporta ningun beneficio
  • http://shoulu.jike.com/spider.html
  • Semager - Crawler para una web semantica
  • http://www.semager.de/blog/semager-bots/
  • BLEXBot - Crawler para una web semantica
  • http://webmeup-crawler.com/
  • AdsBot-Google ignores any default rule, and only
  • honors its own entries
  • MJ12bot - Crawler
  • http://www.majestic12.co.uk/
  • Yahoo Slurp- Crawler
  • http://help.yahoo.com/help/us/ysearch/slurp
  • DotBot- Crawler
  • https://moz.com/researchtools/ose/dotbot
  • Yandex - Crawler
  • https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml
  • BingBot - Crawler
  • http://www.bing.com/bingbot.htm
  • AhrefsBot - Crawler
  • http://ahrefs.com/robot/
  • OrangeBot - Crawler
  • support.orangebot@orange.com
  • Screaming Frog SEO Spider
  • https://www.screamingfrog.co.uk/seo-spider/
  • Megaindex
  • https://megaindex.com/crawler
  • Block MegaIndex.ru
  • Seokicks
  • https://en.seokicks.de/robot.html
  • LTX71
  • http://ltx71.com/
  • Sputnik
  • http://corp.sputnik.ru/webmaster
  • Open Link Profiler
  • http://openlinkprofiler.org/bot
  • spider@seoscanners.net
  • spider@seoscanners.net
  • Linguee
  • http://www.linguee.com/bot
  • SemrushBot
Back to top