businessinsider.com
robots.txt
Robots Exclusion Standard data for businessinsider.com
Resource Scan
Scan Details
| Site Domain | businessinsider.com |
| Base Domain | businessinsider.com |
| Scan Status | Ok |
| Last Scan | 2024-11-09T10:19:25+00:00 |
| Next Scan | 2024-11-16T10:19:25+00:00 |
Last Scan
| Scanned | 2024-11-09T10:19:25+00:00 |
| URL | https://businessinsider.com/robots.txt |
| Redirect | https://www.businessinsider.com/robots.txt |
| Redirect Domain | www.businessinsider.com |
| Redirect Base | businessinsider.com |
| Domain IPs | 151.101.1.171, 151.101.129.171, 151.101.193.171, 151.101.65.171 |
| Redirect IPs | 151.101.1.171, 151.101.129.171, 151.101.193.171, 151.101.65.171 |
| Response IP | 199.232.45.171 |
| Found | Yes |
| Hash | de014b628e9546c801777c20841de196683e2812d2ec315f87b023fa72e6ff20 |
| SimHash | 9b5a484fccdb |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /*?utm_campaign=Monitor& |
| Disallow | /adframe |
| Disallow | /afp$ |
| Disallow | /ajax/ |
| Disallow | /answers$ |
| Disallow | /archives |
| Disallow | /associated-press$ |
| Disallow | /authentication$ |
| Disallow | /author/*/date |
| Disallow | /author/*/mostread |
| Disallow | /categories |
| Disallow | /cms/ |
| Disallow | /comments$ |
| Disallow | /cross-domain$ |
| Disallow | /document/ |
| Disallow | /insider$ |
| Disallow | /partner/ |
| Disallow | /reuters$ |
| Disallow | /reviews/out? |
| Disallow | /s? |
| Disallow | /guides/s? |
| Disallow | /personal-finance/s? |
| Disallow | /track.gif |
| Disallow | /uk$ |
| Disallow | /ws/ |
| Disallow | /business-insider$ |
| Disallow | /*/contributor |
| Disallow | /bi$ |
| Disallow | /news-insider$ |
| Disallow | /*/rss |
| Disallow | /*.rss |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.businessinsider.com/sitemap/latest.xml |
| sitemap | https://www.businessinsider.com/sitemap/google-news.xml |
| sitemap | https://www.businessinsider.com/sitemap/index.xml |
| sitemap | https://www.businessinsider.com/sitemap/landing-pages.xml |
| sitemap | https://feeds.businessinsider.com/custom/tech-sj-sitemap |
| sitemap | https://feeds.businessinsider.com/custom/pfi-sitemap |