Log In Sign Up

foroswebgratis.biz
robots.txt

Robots Exclusion Standard data for foroswebgratis.biz

Resource Scan

Scan Details

Site Domain foroswebgratis.biz
Base Domain foroswebgratis.biz
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-14T12:36:50+00:00
Next Scan 2024-12-13T12:36:50+00:00

Last Successful Scan

Scanned2022-09-07T19:02:39+00:00
URL https://foroswebgratis.biz/robots.txt
Response IP 104.21.92.93
Found Yes
Hash 2abae4a184998bb90a19caeb7db3e9f5d142aea307d449e17d763b80622cf83f
SimHash 93617353846a

Groups

rogerbot
exabot
mj12bot
dotbot
gigabot
ahrefsbot
blackwidow
chinaclaw
custo
disco
download\ demon
ecatch
eirgrabber
emailsiphon
emailwolf
express\ webpictures
extractorpro
eyenetie
flashget
getright
getweb!
go!zilla
go-ahead-got-it
grabnet
grafula
hmview
httrack
image\ stripper
image\ sucker
indy\ library
interget
internet\ ninja
jetcar
joc\ web\ spider
larbin
leechftp
mass\ downloader
midown\ tool
mister\ pix
navroad
nearsite
netants
netspider
net\ vampire
netzip
octopus
offline\ explorer
offline\ navigator
pagegrabber
papa\ foto
pavuk
pcbrowser
realdownload
reget
sitesnagger
smartdownload
superbot
superhttp
surfbot
takeout
teleport\ pro
voideye
web\ image\ collector
web\ sucker
webauto
webcopier
webfetch
webgo\ is
webleacher
webreaper
websauger
website\ extractor
website\ quester
webstripper
webwhacker
webzip
wget
widow
wwwoffle
xaldon\ webspider
zeus

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

exabot

Rule Path
Disallow /

pathdefender

Rule Path
Disallow /

goodzer

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bspider

Rule Path
Disallow /

voltron

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

getintentcrawler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

teoma

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

yahoo-blogs/v3.9

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /php/

wotbox

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

alexabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /
Back to top

Comments

  • Block Gigabot
  • Block AhrefsBot
  • Block YandexBot
  • Block Yandex
  • Block Exabot
  • Block PathDefender
  • Block Goodzer
  • Block MegaIndex.ru
  • Block Ezooms
  • Block Baiduspider
  • Block common cralwers:
  • Some weird backlink checker (webmeup.com)
  • Another backlink crawler
  • Another backlink crawler (https://ahrefs.com/robot)
  • Another one.
Back to top

Warnings

  • 2 invalid lines.
Back to top