bostonphoenix.com
robots.txt
Robots Exclusion Standard data for bostonphoenix.com
Resource Scan
Scan Details
| Site Domain | bostonphoenix.com |
| Base Domain | bostonphoenix.com |
| Scan Status | Ok |
| Last Scan | 2025-05-20T06:13:37+00:00 |
| Next Scan | 2025-05-27T06:13:37+00:00 |
Last Scan
| Scanned | 2025-05-20T06:13:37+00:00 |
| URL | https://bostonphoenix.com/robots.txt |
| Domain IPs | 64.90.49.183 |
| Response IP | 64.90.49.183 |
| Found | Yes |
| Hash | b0600589954198d125cf44fc262e9bf1c3fb0123590b332638ed5f6c0ab7957c |
| SimHash | cbe0c9e34e79 |
Groups
turnitinbot
semrushbot
eyemonit
buck
mediatoolkitbot
seznambot
taboolabot
mail.ru
bidswitchbot
mj12bot
zoombot
seostar
neevabot
zoominfobot
linespider
megaindex.ru
petalbot
| Rule | Path |
|---|---|
| Disallow | / |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
| crawl-delay | 5 |
*
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /waypoint/ |
| Disallow | /BTest/ |
| Disallow | /common_files/ |
| Disallow | /common_images/ |
| Disallow | /includes/ |
| Disallow | /utility/ |
| Disallow | /boston/food_drink/dining/documents/02157961.htm |
| Disallow | /tools/ |
| Disallow | /Tools/ |
| Disallow | /Boston/Topic/ |
| Disallow | /Portland/Topic/ |
| Disallow | /Providence/Topic/ |
| Disallow | /Boston/Concerts/Artists/ |
| Disallow | /Portland/Concerts/Artists/ |
| Disallow | /Providence/Concerts/Artists/ |
| Disallow | /boston/topic/ |
| Disallow | /portland/topic/ |
| Disallow | /providence/topic/ |
| Disallow | /widget/ |
| Disallow | /Widget/ |
| Disallow | /widgets/ |
| Disallow | /Widgets/ |
| Disallow | /%3Dpromos/ |
| Disallow | /classifieds/ |
| Disallow | /Boston/classifieds/ |
| Disallow | /Portland/classifieds/ |
| Disallow | /Providence/classifieds/ |
| Disallow | /community/ |
| Disallow | /Community/ |
| Disallow | /COMMUNITY/POLLS/ |
| Disallow | /COMMUNITY/ |
| Disallow | /COMMUNITY/members/ |
| Disallow | /Blogs/Members/ |
| Disallow | /Adult/Members/ |
| Disallow | /adult/forums/ |
| Disallow | /adult/photos/ |
| Disallow | /adult/Members/ |
| Disallow | /ADULT/Members/ |
| Disallow | /mob/ |
| Disallow | /Mob/ |
| Disallow | /Members/ |
| Disallow | /Polls/Members/ |
| Disallow | /polls/Members/ |
| Disallow | /community/members/ |
| Disallow | /blogs/members/ |
| Disallow | /BLOGS/members/ |
| Disallow | /adult/members/ |
| Disallow | /members/ |
| Disallow | /polls/members/ |
| Disallow | /converged/ |
| Disallow | /Converged/ |
| Disallow | /BLOGS/Converged/ |
| Disallow | /blogs/Converged/ |
| Disallow | /BLOGS/puma/ |
| Disallow | /blogs/puma/ |
| Disallow | /blogs/blogs/converged/ |
| Disallow | /BLOGS/forums |
| Disallow | /blogs/forums |
Warnings
- 3 invalid lines.
Comments