app.box.com
robots.txt
Robots Exclusion Standard data for app.box.com
Resource Scan
Scan Details
| Site Domain | app.box.com |
| Base Domain | box.com |
| Scan Status | Ok |
| Last Scan | 2024-11-06T11:48:08+00:00 |
| Next Scan | 2024-11-20T11:48:08+00:00 |
Last Scan
| Scanned | 2024-11-06T11:48:08+00:00 |
| URL | https://app.box.com/robots.txt |
| Domain IPs | 74.112.186.157 |
| Response IP | 74.112.186.157 |
| Found | Yes |
| Hash | db4c993e908ae407c84bdf3d6d128926070fe7e28ec79303e654e1c5d9ff5a9d |
| SimHash | 3a95f14f84e9 |
Groups
twitterbot
| Rule | Path | Comment |
|---|---|---|
| Disallow | / | - |
| Allow | /representation | Twitter needs to be able to display file thumbnails in social media cards; this prevents those assets from being blocked. |
| Allow | /app-assets | Twitter needs to be able to display the Box logo in social media cards; this prevents those assets from being blocked. |
| Allow | /$ | - |
| Allow | /sitemap.xml$ | - |
| Allow | /login | - |
| Allow | /simpleshare | - |
| Allow | /services | - |
| Allow | /reset | - |
| Allow | /signup | - |
| Allow | /cdn | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /sites | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /platform | Platform static assets (Box Elements) share a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /s/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /shared/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /v/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /app-center | - |
| Allow | /integrations | - |
| Allow | /enduser | - |
| Allow | /fonts | - |
| Allow | /_assets/img/favicons | - |
| Allow | /app-api/split-proxy | - |
| Allow | /app-gallery-v2/api | - |
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | / | - |
| Disallow | /signup/collablink/ | - |
| Allow | /$ | - |
| Allow | /sitemap.xml$ | - |
| Allow | /login | - |
| Allow | /simpleshare | - |
| Allow | /services | - |
| Allow | /reset | - |
| Allow | /signup | - |
| Allow | /cdn | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /sites | The marketing site shares a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /platform | Platform static assets (Box Elements) share a CDN with the webapp; this prevents those assets from being blocked. |
| Allow | /s/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /shared/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /v/ | The "noindex" robots directive is only respected if these pages are crawlable |
| Allow | /app-center | - |
| Allow | /integrations | - |
| Allow | /enduser | - |
| Allow | /fonts | - |
| Allow | /_assets/img/favicons | - |
| Allow | /app-api/split-proxy | - |
| Allow | /app-gallery-v2/api | - |