brownbook.net
robots.txt
Robots Exclusion Standard data for brownbook.net
Resource Scan
Scan Details
| Site Domain | brownbook.net |
| Base Domain | brownbook.net |
| Scan Status | Ok |
| Last Scan | 2024-09-20T05:48:13+00:00 |
| Next Scan | 2024-09-27T05:48:13+00:00 |
Last Scan
| Scanned | 2024-09-20T05:48:13+00:00 |
| URL | https://brownbook.net/robots.txt |
| Redirect | https://www.brownbook.net/robots.txt |
| Redirect Domain | www.brownbook.net |
| Redirect Base | brownbook.net |
| Domain IPs | 104.22.26.166, 104.22.27.166, 172.67.4.72, 2606:4700:10::6816:1aa6, 2606:4700:10::6816:1ba6, 2606:4700:10::ac43:448 |
| Redirect IPs | 104.22.26.166, 104.22.27.166, 172.67.4.72, 2606:4700:10::6816:1aa6, 2606:4700:10::6816:1ba6, 2606:4700:10::ac43:448 |
| Response IP | 172.67.4.72 |
| Found | Yes |
| Hash | 0bb7f28600b2a4d888b67a83d977303f913b56c6edc52606e0c60abbb978f386 |
| SimHash | 32bd9b59f795 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /aj/ |
| Disallow | /ajax/ |
| Disallow | /account/* |
| Disallow | /images/ |
| Disallow | /includes/ |
| Disallow | /payment/ |
| Disallow | /user/login/ |
| Disallow | /user/register/ |
| Disallow | /user/forgotten_password/ |
| Disallow | /user/reset_password/ |
| Disallow | /user/suspended/ |
| Disallow | /review/view/ |
| Disallow | /review/add/ |
| Disallow | /review/confirm_add/ |
| Disallow | /review/reports/ |
| Disallow | /review/vid_popup/ |
| Disallow | /review/invite/ |
| Disallow | /business/edit/ |
| Disallow | /business/add/ |
| Disallow | /business/flag/ |
| Disallow | /business/emailme/ |
| Disallow | /business/emailanother/ |
| Disallow | /business/reports/ |
| Disallow | /business/vcard/ |
| Disallow | /business/map/ |
| Disallow | /business/claim/ |
| Disallow | /dispute/ |
| Disallow | /report/ |
| Disallow | /businesses/ |
| Disallow | */claim |
| Disallow | /business/*/*/claim |
| Disallow | /user-profile/* |
| Disallow | /countries/* |
| Disallow | /countries/*/categories |
| Disallow | /countries/*/states |
| Disallow | /countries/*/states/* |
| Disallow | /countries/*/states/*/cities |
| Disallow | /countries/*/states/*/cities/* |
| Disallow | /business/*/*/preview |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://api.brownbook.net/sitemap.xml |
Comments