Log In Sign Up

gazetvandaag.be
robots.txt

Robots Exclusion Standard data for gazetvandaag.be

Resource Scan

Scan Details

Site Domain gazetvandaag.be
Base Domain gazetvandaag.be
Scan Status Ok
Last Scan2024-11-10T03:48:55+00:00
Next Scan 2024-11-17T03:48:55+00:00

Last Scan

Scanned2024-11-10T03:48:55+00:00
URL https://gazetvandaag.be/robots.txt
Domain IPs 185.104.28.21
Response IP 185.104.28.21
Found Yes
Hash c3650d105466e0e380b3dd9c675a1422f4e84f152cb1c510cfbfa61c4f08fcc9
SimHash faf27db0dcf1

Groups

*

Rule Path
Disallow /images
Disallow /templates

Other Records

Field Value
crawl-delay 2

yahoo! slurp

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

xenu’s

Rule Path
Disallow /

xenu’s link sleuth 1.1c

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

linkfluence

Rule Path
Disallow /

veooz

Rule Path
Disallow /

veoozbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mediawords

Rule Path
Disallow /

spbot

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

deusu

Rule Path
Disallow /

feedlybot/1.0

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

toweyabot

Rule Path
Disallow /

toweya.com bot

Rule Path
Disallow /

python/3.5

Rule Path
Disallow /

spbot

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

idbot

Rule Path
Disallow /

idbot/1.1

Rule Path
Disallow /

addthis

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

uipbot

Rule Path
Disallow /

umbot-ln/1.0

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

laserlikebot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

um-fc/1.0

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

buckyohare

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

yandexmobilebot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

expo9

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

yandexmobilebot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

expo9

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

lcc

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://gazetvandaag.be/sitemap.xml
sitemap https://gazetvandaag.be/articles.xml
Back to top

Warnings

  • 2 invalid lines.
Back to top