gendou.com
robots.txt
Robots Exclusion Standard data for gendou.com
Resource Scan
Scan Details
| Site Domain | gendou.com |
| Base Domain | gendou.com |
| Scan Status | Ok |
| Last Scan | 2024-06-02T20:42:06+00:00 |
| Next Scan | 2024-07-02T20:42:06+00:00 |
Last Scan
| Scanned | 2024-06-02T20:42:06+00:00 |
| URL | http://gendou.com/robots.txt |
| Domain IPs | 208.97.136.254 |
| Response IP | 208.97.136.254 |
| Found | Yes |
| Hash | 2c0da1d295f13dd3341cac5d8fb5efbe9abf18b3c2cf50d3ff46df4b0feaee24 |
| SimHash | f55ed48343b5 |
Groups
apachebench
da
dap
digext
dotbot
download master
e-societyrobot
exabot
fdm
flashget
getleft
getright
gigabot
heritrix
httrack
ia_archiver
ida
leechget
libwww-perl
msnbot
msrbot
omniexplorer_bot
puxarapido
riddler
skreemrbot
stardownloader
webcopier
yandex
yeti
yodaobot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /bot-trap.php |
| Disallow | /images/ |
| Disallow | /amusic/dl.php |
| Disallow | /amusic/rate.php |
| Disallow | /forum/search.php |
| Disallow | /downloads/ |
| Disallow | /uploads/ |
| Disallow | /amusic/my-request.php |
| Disallow | /*sort%3D |
| Disallow | /*cols%3D |
| Disallow | /*ent%3D |
| Disallow | /*back%3D |
| Disallow | /*rev%3D |
| Disallow | /*do%3D |
| Disallow | /forum/user.php |
| Disallow | /forum/sticky.php |
| Disallow | /forum/lock.php |
| Disallow | /q/lyrics.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |