nwlab.net
robots.txt
Robots Exclusion Standard data for nwlab.net
Resource Scan
Scan Details
| Site Domain | nwlab.net |
| Base Domain | nwlab.net |
| Scan Status | Ok |
| Last Scan | 2024-09-23T11:46:17+00:00 |
| Next Scan | 2024-09-30T11:46:17+00:00 |
Last Scan
| Scanned | 2024-09-23T11:46:17+00:00 |
| URL | https://www.nwlab.net/robots.txt |
| Domain IPs | 2a01:4f8:c010:3112::1, 5.75.235.30 |
| Response IP | 5.75.235.30 |
| Found | Yes |
| Hash | ca67542e7e743b7d4347971c8bee818c740d0219c98b6a0125552aabed887428 |
| SimHash | a4d658b4c401 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /test |
| Disallow | /cgi-bin |
| Disallow | /hidden |
| Disallow | /forum |
| Disallow | /forBotsOnly |
googlebot
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /test |
| Disallow | /cgi-bin |
| Disallow | /hidden |
| Disallow | /forum |
| Disallow | /forBotsOnly |
slurp
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /test |
| Disallow | /cgi-bin |
| Disallow | /hidden |
| Disallow | /forum |
| Disallow | /forBotsOnly |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 200 |
msnbot
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /test |
| Disallow | /cgi-bin |
| Disallow | /hidden |
| Disallow | /forum |
| Disallow | /forBotsOnly |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 200 |
laycat
jobs.de-robot
eurobot
gigabot
speedy
surveybot_ignoreip
surveybot
heise-it-markt-crawler
sitecheck.internetseer.com
zealbot
msiecrawler
sitesnagger
webstripper
webcopier
fetch
fyberspider
offline explorer
teleport
teleportpro
webzip
linko
httrack
microsoft.url.control
xenu
larbin
libwww
zyborg
download ninja
wget
grub-client
webreaper
scoutjet
jobroboter
gonzo*
envolk
psbot
crawly
baiduspider
blogpulselive
searchlink
woriobot
snapbot
flatlandbot
infometrics-bot
exabot
kalooga
mj12bot
youdaobot
touche
sosospider
tasapspider
euripbot
iearthworm
mail.ru
echobot
dotbot
iccrawler
ruky-bot
nebullabot
netluchs
ocelli
| Rule | Path |
|---|---|
| Disallow | / |
Comments