margo24.pl
robots.txt
Robots Exclusion Standard data for margo24.pl
Resource Scan
Scan Details
| Site Domain | margo24.pl |
| Base Domain | margo24.pl |
| Scan Status | Ok |
| Last Scan | 2025-12-12T18:38:00+00:00 |
| Next Scan | 2026-01-11T18:38:00+00:00 |
Last Scan
| Scanned | 2025-12-12T18:38:00+00:00 |
| URL | https://margo24.pl/robots.txt |
| Domain IPs | 104.26.2.101, 104.26.3.101, 172.67.75.8, 2606:4700:20::681a:265, 2606:4700:20::681a:365, 2606:4700:20::ac43:4b08 |
| Response IP | 172.67.75.8 |
| Found | Yes |
| Hash | 91bd8d8594b0c97a68c5551384da40e14927a26d52ca660ef2069ff1e3ec83a2 |
| SimHash | 329853c3e962 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | */negotiation-form?sku= |
| Disallow | */privacy-policy |
| Disallow | */review/ |
| Disallow | */tag/ |
| Disallow | */home |
| Disallow | */index.php |
| Disallow | */wishlist |
| Disallow | */checkout |
| Disallow | */cart$ |
| Disallow | *sortDirection%3D |
| Disallow | *sortAttribute%3D |
| Disallow | *gridViewType%3D |
| Disallow | *pagesize%3D |
| Disallow | *%3A |
| Allow | *page%3A |
| Disallow | *.html/1000 |
chatgpt-user
oai-searchbot
claudebot
perplexitybot
ccbot
google-extended
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | */checkout |
| Disallow | */cart |
| Disallow | */account |
| Disallow | */wishlist |
screaming frog
semrushbot-si
siteauditbot
semrushbot-ba
semrushbot-coub
semrushbot-ct
semrushbot
vbseo
seoengworldbot
semrushbot-bm
semrushbot-swa
rogerbot
splitsignalbot
mj12bot
xenus link sleuth 1.1c
xenus
ubicrawler
doc
zao
sitecheck.internetseer.com
zealbot
omniexplorer_bot
freefind
becomebot
nutch
askjeeves
teoma
jeeves
stanford
naverbot
dumbot
hatena antenna
grub-client
grub
looksmart
webzip
larbin
b2w/0.1
zyborg
libwww
download ninja
copernic
psbot
python-urllib
netmechanic
url_spider_pro
cherrypicker
emailcollector
emailsiphon
webbandit
emailwolf
extractorpro
copyrightcheck
crescent
sitesnagger
prowebwalker
cheesebot
lnspiderguy
ia_archiver
archive.org_bot
k2spider
npbot
webreaper
alexibot
teleport
teleportpro
stanford comp sci
miixpc
telesoft
website quester
moget/2.1
webzip/4.0
webstripper
linko
httrack
microsoft.url.control
websauger
webcopier
fetch
netants
mister pix
webauto
thenomad
www-collector-e
rma
libweb/clshttp
asterias
httplib
turingos
spanner
infonavirobot
harvest/1.5
bullseye/1.0
mozilla/4.0 (compatible; bullseye; windows 95)
crescent internet toolpak http ole control v.1.0
cherrypickerse/1.0
cherrypickerelite/1.0
webbandit/3.50
nicerspro
microsoft url control - 5.01.4511
dittospyder
foobot
webmasterworldforumbot
spankbot
botalot
lwp-trivial/1.34
lwp-trivial
http://www.webmasterworld.com bot
bunnyslippers
microsoft url control - 6.00.8169
urly warning
wget1.6
wget/1.5.3
wget
linkwalker
cosmos
moget
hloader
humanlinks
linkextractorpro
offline explorer
mata hari
lexibot
web image collector
the intraformant
true_robot/1.0
true_robot
blowfish/1.0
http://www.searchengineworld.com bot
jennybot
miixpc/4.2
builtbottough
propowerbot/2.14
backdoorbot/1.0
tocrawl/urldispatcher
webenhancer
suzuran
vci webviewer vci webviewer win32
vci
szukacz/1.4
openfind data gathere
openfind
zeus
repomonkey bait & tackle/v1.01
repomonkey
microsoft url control
openbot
url control
zeus link scout
zeus 32297 webster pro v2.9 win32
webster pro
linkscan/8.1a unix
keyword density/0.9
kenjin spider
iron33/1.0.2
bookmark search tool
getright/4.2
fairad client
gaisbot
aqua_products
radiation retriever 1.1
webmasterworld extractor
flaming attackbot
oracle ultra search
msiecrawler
perman
searchpreview
sootle
es
enterprise_search/1.0
enterprise_search
blexbot
dotbot
eccp/1.0
eccp
proximic
easouspider
grapeshot
siteexplorer
sogou spider
mail.ru
trendictionbot
baiduspider
voltron
sistrix
updownerbot
meanpathbot
exabot
scoutjet
contextad
spbot
snappreviewbot
xovibot
zumbot
searchmetricsbot
crazywebcrawler-spider
voilabot
wotbox
crystalsemanticsbot
smtbot
musobot
seokicks-robot
frcrawler
turnitinbot
ca-crawler
nekstbot
nerdybot
mixrankbot
obot
advbot
memorybot
linkpadbot
yacybot
haosouspider
fatbot
cliqzbot
dnyzbot
yetibot
bubing
nekstbot
findxbot
petalbot
| Rule | Path |
|---|---|
| Disallow | / |
Warnings
- 2 invalid lines.
Comments