poemhunter.com
robots.txt
Robots Exclusion Standard data for poemhunter.com
Resource Scan
Scan Details
| Site Domain | poemhunter.com |
| Base Domain | poemhunter.com |
| Scan Status | Ok |
| Last Scan | 2024-05-21T20:37:24+00:00 |
| Next Scan | 2024-05-28T20:37:24+00:00 |
Last Scan
| Scanned | 2024-05-21T20:37:24+00:00 |
| URL | https://poemhunter.com/robots.txt |
| Redirect | https://www.poemhunter.com/robots.txt |
| Redirect Domain | www.poemhunter.com |
| Redirect Base | poemhunter.com |
| Domain IPs | 95.168.183.149 |
| Redirect IPs | 51.161.197.225, 51.161.209.128, 51.79.231.108 |
| Response IP | 51.161.197.225 |
| Found | Yes |
| Hash | 2ff01dc5e94d7683f695af7b3c4429ea1222e055bf443c45b4ad65c2fe1eedd1 |
| SimHash | fc5b9d135d19 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Allow | /amp/poem/poet-*/ |
| Allow | /poem/poet-*/ |
| Allow | /poet-*/ |
| Allow | /poems/*-*/short |
| Disallow | /poemhunter/show.asp* |
| Disallow | /poemhunter/msg.asp* |
| Disallow | /Ajax/ |
| Disallow | /Profile/ |
| Disallow | /Activity/ |
| Disallow | /member/ |
| Disallow | /poem-amp/ |
| Disallow | /best-poem/ |
| Disallow | */comments/* |
| Disallow | /quotations/ |
| Disallow | /errors/ |
| Disallow | /explore/poems/ |
| Disallow | /poems/new-poems/page* |
| Disallow | /poems/top-member-poems/page* |
| Disallow | /poems/classical-poems/page* |
| Disallow | /top-100-poems/ |
| Disallow | /poems/best-poems/member-poets/page* |
| Disallow | /poems/best-poems/classical-poets/page* |
| Disallow | /explore/poets/ |
| Disallow | /poets/new-poets/page* |
| Disallow | /poets/popular-member-poets/page* |
| Disallow | /poets/best-poets/member-poets/page* |
| Disallow | /poets/classics-poets/page* |
| Disallow | /poets/best-poets/classic-poets/page* |
| Disallow | *?m=0 |
| Disallow | /ebooks/ |
| Disallow | /poets/?page* |
| Disallow | /poems/*/page-* |
| Disallow | /poems/*-* |
| Disallow | *?topic=* |
| Disallow | /poem/*/page-* |
| Disallow | /*/poet-*/ |
| Disallow | /search/ |
| Disallow | /*/following/ |
| Disallow | /*/followers/ |
| Disallow | /*/activity/ |
| Disallow | /*/stats/ |
| Disallow | /*/biography/ |
| Disallow | /*/quotations/ |
| Disallow | /amp/ |
| Disallow | /best-poems/*/*/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.poemhunter.com/sitemap_poems_famous.asp |
| sitemap | https://www.poemhunter.com/sitemap_poets_famous.asp |
| sitemap | https://www.poemhunter.com/sitemap/new-poems.asp |
| sitemap | https://www.poemhunter.com/sitemap/new-poets.asp |
| sitemap | https://www.poemhunter.com/sitemaps/poem_topics.asp |