bengo4.com
robots.txt
Robots Exclusion Standard data for bengo4.com
Resource Scan
Scan Details
| Site Domain | bengo4.com |
| Base Domain | bengo4.com |
| Scan Status | Ok |
| Last Scan | 2024-05-10T17:20:23+00:00 |
| Next Scan | 2024-05-17T17:20:23+00:00 |
Last Scan
| Scanned | 2024-05-10T17:20:23+00:00 |
| URL | https://bengo4.com/robots.txt |
| Redirect | https://www.bengo4.com:443/robots.txt |
| Redirect Domain | www.bengo4.com |
| Redirect Base | bengo4.com |
| Domain IPs | 35.72.110.180, 35.78.110.51, 52.194.43.253 |
| Redirect IPs | 23.52.171.121, 23.52.171.136 |
| Response IP | 23.202.33.210 |
| Found | Yes |
| Hash | e02950720e9c84b5db9210172aa35319a72764ac71c8f48ab7e23e6ad653d4b3 |
| SimHash | cb2e9441dd31 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /74013581/ |
| Disallow | /api/ |
| Disallow | /c_3/api/ |
| Disallow | /fudosan/api/ |
| Allow | /api/bbs/rss |
| Allow | /api/img/ |
| Disallow | /bbs/vote/ |
| Disallow | /js/robotsDeny/ |
| Disallow | /js/pc/robotsDeny/ |
| Disallow | /js/sp/robotsDeny/ |
| Disallow | /js/test/robotsDeny/ |
| Disallow | /sp/search/result/*/ |
| Disallow | /sp/user/ |
| Disallow | /result/ |
| Disallow | /topics/jump/ |
| Disallow | /special/ |
| Disallow | /module/ |
| Disallow | /voice/* |
| Allow | /voice/$ |
| Disallow | /sp/voice/* |
| Allow | /sp/voice/$ |
| Disallow | /lawyer/private/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.bengo4.com/topics/feed/googlenews |
| sitemap | https://www.bengo4.com/times/sitemap.xml |