sim-cms-develop-columnists.azurewebsites.net
robots.txt
Robots Exclusion Standard data for sim-cms-develop-columnists.azurewebsites.net
Resource Scan
Scan Details
| Site Domain | sim-cms-develop-columnists.azurewebsites.net |
| Base Domain | sim-cms-develop-columnists.azurewebsites.net |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-09-09T03:24:04+00:00 |
| Next Scan | 2024-12-08T03:24:04+00:00 |
Last Successful Scan
| Scanned | 2022-10-20T08:45:52+00:00 |
| URL | https://sim-cms-develop-columnists.azurewebsites.net/robots.txt |
| Response IP | 40.79.65.200 |
| Found | Yes |
| Hash | 92a4723cbca4747e7bd4ca5da4bbf7105e814a5d4a64979c60721f638f40802c |
| SimHash | 310dd090d691 |
Groups
googlebot
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
msnbot
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
architextspider
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
slurp.so/1.0
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
slurp/2.0j
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
slurp/2.0-kite-hourly
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
slurp/2.0-owlweekly
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
slurp/3.0-au
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
teoma
| Product | Comment |
|---|---|
| teoma | ASK.com |
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
scooter
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
lycos_spider_(t-rex)
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
infoseek sidewinder/9.0
| Rule | Path |
|---|---|
| Disallow | /siteinfo/ |
| Disallow | /images/ |
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
*
| Rule | Path |
|---|---|
| Disallow | /ecards/ |
| Disallow | /*.axd$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Comments