box.net
robots.txt
Robots Exclusion Standard data for box.net
Resource Scan
Scan Details
| Site Domain | box.net |
| Base Domain | box.net |
| Scan Status | Ok |
| Last Scan | 2024-09-23T15:22:56+00:00 |
| Next Scan | 2024-10-07T15:22:56+00:00 |
Last Scan
| Scanned | 2024-09-23T15:22:56+00:00 |
| URL | https://box.net/robots.txt |
| Domain IPs | 74.112.186.157 |
| Response IP | 74.112.186.157 |
| Found | Yes |
| Hash | eb13c0ca5f3cfb1a8f00372ca88828306a24980d2e37ee11b1e99c880780da77 |
| SimHash | 3a95f14784e9 |
Groups
twitterbot
| Rule | Path | Comment |
|---|---|---|
| Disallow | / | - |
| Allow | /representation | Twitter needs to be able to display file thumbnails in social media cards; this prevents those assets from being blocked. |
| Allow | /app-assets | Twitter needs to be able to display the Box logo in social media cards; this prevents those assets from being blocked. |
| Allow | /$ | - |
| Allow | /sitemap.xml$ | - |
| Allow | /login | - |
| Allow | /simpleshare | - |
| Allow | /services | - |
| Allow | /reset | - |
| Allow | /signup | - |
| Allow | /cdn | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /sites | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /platform | Platform static assets (Box Elements) share a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /s/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /shared/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /v/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /app-center | - |
| Allow | /enduser | - |
| Allow | /fonts | - |
| Allow | /_assets/img/favicons | - |
| Allow | /app-api/split-proxy | - |
| Allow | /app-gallery-v2/api | - |
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | / | - |
| Disallow | /signup/collablink/ | - |
| Allow | /$ | - |
| Allow | /sitemap.xml$ | - |
| Allow | /login | - |
| Allow | /simpleshare | - |
| Allow | /services | - |
| Allow | /reset | - |
| Allow | /signup | - |
| Allow | /cdn | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /sites | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /platform | Platform static assets (Box Elements) share a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /s/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /shared/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /v/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /app-center | - |
| Allow | /enduser | - |
| Allow | /fonts | - |
| Allow | /_assets/img/favicons | - |
| Allow | /app-api/split-proxy | - |
| Allow | /app-gallery-v2/api | - |