globe.com
robots.txt
Robots Exclusion Standard data for globe.com
Resource Scan
Scan Details
| Site Domain | globe.com |
| Base Domain | globe.com |
| Scan Status | Ok |
| Last Scan | 2024-04-29T07:55:39+00:00 |
| Next Scan | 2024-05-06T07:55:39+00:00 |
Last Scan
| Scanned | 2024-04-29T07:55:39+00:00 |
| URL | https://globe.com/robots.txt |
| Redirect | https://www.bostonglobe.com/robots.txt |
| Redirect Domain | www.bostonglobe.com |
| Redirect Base | bostonglobe.com |
| Domain IPs | 104.18.14.134, 104.18.15.134, 2606:4700::6812:e86, 2606:4700::6812:f86 |
| Redirect IPs | 23.209.46.18, 23.209.46.25, 2600:1413:b000:14::b857:c149, 2600:1413:b000:14::b857:c157 |
| Response IP | 42.99.140.211 |
| Found | Yes |
| Hash | 7120c6ef67e1864f7bdbb53e4755dce4115f1fa6cb3784e4a8ae30ea367969e6 |
| SimHash | 4ac4105da7f7 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /insiders |
| Disallow | /Statistics |
| Disallow | /metro/2015/05/24/pakistani-employee-sues-onebeacon-claiming-harassment/yScUj6mmmU2IwM8hLSDhTN/story.html |
| Disallow | /fragment-global-A8zF2y/ |
| Disallow | /fragment-spt-global-A8zF2y/ |
| Disallow | /trendingbar/ |
| Disallow | /opinion-fragment/ |
| Disallow | /magazine-fragment/ |
| Disallow | /marijuana-fragment/ |
| Disallow | /stat-fragment/ |
| Disallow | /rh-cong3-fragment/ |
| Disallow | /iowa-2020-fragment_stage_only/ |
| Disallow | /am-fragment_stage_only/ |
| Disallow | /rh-cong3-fragment_stage_only/ |
| Disallow | /rh-fragment2_stage_only/ |
| Disallow | /rh-fragment4_stage_only/ |
| Disallow | /ys-fragment_stage_only/ |
| Disallow | /ys-fragment-2_stage_only/ |
| Disallow | /ys-fragment-3_stage_only/ |
| Disallow | /ys-fragment-4_stage_only/ |
| Disallow | /demo-starter-master/ |
| Disallow | /overlineTest/ |
| Disallow | /sports/xyz123/ |
| Disallow | /about/help/terms/ |
| Disallow | /todays-paper/arts-lifestyle/ |
| Disallow | /todays-paper/business/ |
| Disallow | /todays-paper/comfort-zone/ |
| Disallow | /todays-paper/opinion/ |
| Disallow | /todays-paper/ideas/ |
| Disallow | /todays-paper/magazine/ |
| Disallow | /todays-paper/metro/ |
| Disallow | /todays-paper/obituaries/ |
| Disallow | /todays-paper/real-estate/ |
| Disallow | /todays-paper/sports/ |
| Disallow | /todays-paper/nation/ |
| Disallow | /todays-paper/travel/ |
| Disallow | /todays-paper/wednesday-food/ |
| Disallow | /todays-paper/weekend/ |
| Disallow | /todays-paper/world/ |
| Allow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | /arc/outboundfeeds/sitemap/?outputType=xml |
| sitemap | /arc/outboundfeeds/news-sitemap/?outputType=xml |
| sitemap | /sitemap.xml |
Comments