dlsite.com
robots.txt
Robots Exclusion Standard data for dlsite.com
Resource Scan
Scan Details
Site Domain | dlsite.com |
Base Domain | dlsite.com |
Scan Status | Ok |
Last Scan | 2024-05-31T11:43:56+00:00 |
Next Scan | 2024-06-30T11:43:56+00:00 |
Last Scan
Scanned | 2024-05-31T11:43:56+00:00 |
URL | https://dlsite.com/robots.txt |
Redirect | http://www.dlsite.com/robots.txt |
Redirect Domain | www.dlsite.com |
Redirect Base | dlsite.com |
Domain IPs | 3.114.240.192, 35.78.143.52 |
Redirect IPs | 18.155.68.29, 18.155.68.83, 18.155.68.87, 18.155.68.91 |
Response IP | 18.155.68.83 |
Found | Yes |
Hash | 035aac72cf71864cbea242d9bf029f63f196dcac36fa3aa605d9f5ad55fc28ff |
SimHash | 6f3486604b22 |
Groups
bingbot
Rule | Path |
---|---|
Disallow | */fsr/%3D/*/per_page/*/page/* |
Allow | / |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
*
Rule | Path |
---|---|
Disallow | /*module |
Disallow | /*action |
Disallow | /*adultcheck |
Disallow | /error2.html |
Disallow | /*mypage |
Disallow | /*simura |
Disallow | /*search/result |
Disallow | /*maker/profile/dlblog/content |
Disallow | /hana/* |
Disallow | /booksl/* |
Disallow | /pro2/* |
Disallow | */dealer/* |
Disallow | */load/recommend/* |
Disallow | */load/review/* |
Disallow | */article/list/* |
Disallow | */work/image/sample/* |
Disallow | /js/adultcheck.js |
Disallow | */load/favorite/product |
Disallow | */load/not/interest |
Disallow | */load/bought/product |
Disallow | */user/product/smmary/%3D/products/ |
Disallow | */announce/summary |
Disallow | /circle/profile/ |
Disallow | */author/list/ |
Disallow | /eisys/blog/auth.php |
Disallow | /work/%3D/product_id/ |
Disallow | /work/reviewlist/ |
Disallow | /popup/ |
Allow | / |
Allow | */mypage/wishlist |
Allow | */mypage/followlist |
Allow | */load/recommend/%3D/* |
Other Records
Field | Value |
---|---|
sitemap | https://www.dlsite.com/maniax/sitemap/index/xml |
sitemap | https://www.dlsite.com/home/sitemap/index/xml |
sitemap | https://www.dlsite.com/books/sitemap/index/xml |
sitemap | https://www.dlsite.com/comic/sitemap/index/xml |
sitemap | https://www.dlsite.com/pro/sitemap/index/xml |
sitemap | https://www.dlsite.com/soft/sitemap/index/xml |
sitemap | https://www.dlsite.com/girls/sitemap/index/xml |
sitemap | https://www.dlsite.com/girls-pro/sitemap/index/xml |
sitemap | https://www.dlsite.com/girls-drama/sitemap/index/xml |
sitemap | https://www.dlsite.com/bl/sitemap/index/xml |
sitemap | https://www.dlsite.com/bl-pro/sitemap/index/xml |
sitemap | https://www.dlsite.com/bl-drama/sitemap/index/xml |
Comments