gymfed.be
robots.txt
Robots Exclusion Standard data for gymfed.be
Resource Scan
Scan Details
| Site Domain | gymfed.be |
| Base Domain | gymfed.be |
| Scan Status | Ok |
| Last Scan | 2024-05-24T00:10:52+00:00 |
| Next Scan | 2024-06-23T00:10:52+00:00 |
Last Scan
| Scanned | 2024-05-24T00:10:52+00:00 |
| URL | https://gymfed.be/robots.txt |
| Redirect | https://www.gymfed.be/robots.txt |
| Redirect Domain | www.gymfed.be |
| Redirect Base | gymfed.be |
| Domain IPs | 178.208.36.107 |
| Redirect IPs | 178.208.36.107 |
| Response IP | 178.208.36.107 |
| Found | Yes |
| Hash | 06daabc67950e49b94f8c3dc233bebc2812bfe91ce1156e18e045b4383af780e |
| SimHash | 783a62c4fe19 |
Groups
crawler4j
crawler4j (http://code.google.com/p/crawler4j/)
curious george - www.analyticsseo.com/crawler
curious george - www.analyticsseo.com
megaindex.ru/2.0
megaindex.ru/
mozilla/5.0 (compatible; megaindex.ru/2.0; +https://www.megaindex.ru/?tab=linkanalyze)
mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
mj12
mozilla/5.0 (compatible; steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
steeler
microsoft-webdav-miniredir/6.1.7601
mozilla/5.0 (compatible; findxbot/1.0; +http://www.findxbot.com)
findxbot
mozilla/5.0 (compatible; seznambot/3.2; +http://fulltext.sblog.cz/)
seznambot
seznam
mozilla/4.0 (compatible; vagabondo/4.0; http://webagent.wise-guys.nl/)
mozilla/4.0 (compatible; vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
vagabondo
jakarta commons-httpclient/3.0.1
lwp::simple/5.822
mozilla/5.0 (compatible; smtbot/1.0; +http://www.similartech.com/smtbot)
smtbot
mozilla/5.0 (compatible; memorybot/1.22.56 +http://internetmemory.org/en/)
memorybot
mozilla/5.0 (compatible; smrjbot/0.0.20)
smrjbot
mozilla/5.0 (compatible; spbot/4.4.2; +http://openlinkprofiler.org/bot )
spbot
mj12bot
mj12bot/v1.4.8
mozilla/5.0 (compatible; mj12bot/v1.4.8; http://mj12bot.com/)
baiduspider
exabot
yandex
yandexbot
bspider
panscient.com
bubing
is alpha/nutch-1.1
tineye
ltx71
dotbot
proximic
grapeshot
grapeshotcrawler/2.0
zoombot
qwantify
mauibot
cliqzbot
contxbot
ias_crawler
semrushbot
petalbot
blexbot
daum
ahrefsbot
dataforseobot
blexbot
yioopbot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /members |
| Disallow | /index.html |
| Disallow | /application |
| Disallow | /bonfire |
| Disallow | /public |
Warnings
- 2 invalid lines.
Comments