teq.queensland.com
robots.txt
Robots Exclusion Standard data for teq.queensland.com
Resource Scan
Scan Details
| Site Domain | teq.queensland.com |
| Base Domain | queensland.com |
| Scan Status | Ok |
| Last Scan | 2024-06-01T11:22:08+00:00 |
| Next Scan | 2024-06-15T11:22:08+00:00 |
Last Scan
| Scanned | 2024-06-01T11:22:08+00:00 |
| URL | https://teq.queensland.com/robots.txt |
| Domain IPs | 13.33.30.125, 13.33.30.32, 13.33.30.60, 13.33.30.84 |
| Response IP | 13.33.30.60 |
| Found | Yes |
| Hash | 8e020ada7c13164a27e80e35cb4d0c55274e0c635732edbb7091e3754e7e636e |
| SimHash | 6215945146b4 |
Groups
googlebot
googlebot-image
googlebot-mobile
msnbot
psbot
slurp
yahoo-mmcrawler
yahoo-blogs
baiduspider
baiduspider-image
yandex
teoma
twiceler
gigabot
scrubby
robozilla
bingbot
| Rule | Path | Comment |
|---|---|---|
| Disallow | /App_Browsers/ | - |
| Disallow | /App_Config/ | - |
| Disallow | /App_Data/ | - |
| Disallow | /bin/ | - |
| Disallow | /ClearScript.V8/ | - |
| Disallow | /Content/ | - |
| Disallow | /data/ | - |
| Disallow | /DeployItems/ | - |
| Disallow | /layouts/ | - |
| Disallow | /Properties/ | - |
| Disallow | /Scripts/ | - |
| Disallow | /SearchIndexes/ | - |
| Disallow | /sitecore/ | this one stops indexing of all global content and content in other site tree |
| Disallow | /sitecore%20modules/ | - |
| Disallow | /sitecore_files/ | - |
| Disallow | /upload/ | - |
| Disallow | /Utils/ | - |
| Disallow | /Views/ | - |
| Disallow | /xsl/ | - |
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /App_Browsers/ | - |
| Disallow | /App_Config/ | - |
| Disallow | /App_Data/ | - |
| Disallow | /bin/ | - |
| Disallow | /ClearScript.V8/ | - |
| Disallow | /Content/ | - |
| Disallow | /data/ | - |
| Disallow | /DeployItems/ | - |
| Disallow | /layouts/ | - |
| Disallow | /Properties/ | - |
| Disallow | /Scripts/ | - |
| Disallow | /SearchIndexes/ | - |
| Disallow | /sitecore/ | this one stops indexing of all global content and content in other site tree |
| Disallow | /sitecore%20modules/ | - |
| Disallow | /sitecore_files/ | - |
| Disallow | /upload/ | - |
| Disallow | /Utils/ | - |
| Disallow | /Views/ | - |
| Disallow | /xsl/ | - |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Warnings
- `host` is not a known field.
Comments