dwgdistribution.com
robots.txt
Robots Exclusion Standard data for dwgdistribution.com
Resource Scan
Scan Details
| Site Domain | dwgdistribution.com |
| Base Domain | dwgdistribution.com |
| Scan Status | Ok |
| Last Scan | 2024-11-07T23:36:41+00:00 |
| Next Scan | 2024-12-07T23:36:41+00:00 |
Last Scan
| Scanned | 2024-11-07T23:36:41+00:00 |
| URL | https://dwgdistribution.com/robots.txt |
| Redirect | https://dwg.us/robots.txt |
| Redirect Domain | dwg.us |
| Redirect Base | dwg.us |
| Domain IPs | 162.216.5.39 |
| Redirect IPs | 162.216.5.42 |
| Response IP | 162.216.5.42 |
| Found | Yes |
| Hash | 123a2174bd11c820d73286b3e533aaaf9e47eb544e718ea2b1c2277837604255 |
| SimHash | 2235d4594f10 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /ecommercedev/ |
| Disallow | /ecommerce/dwgadmin12/ |
| Disallow | /ecommerce/pc/search.asp |
| Disallow | /ecommerce/pc/showsearchresults.asp |
| Disallow | /v5store/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
amazonbot
| Product | Comment |
|---|---|
| amazonbot | Amazon's user agent |
| Rule | Path | Comment |
|---|---|---|
| Disallow | / | disallow this directory |
facebookbot
No rules defined. All paths allowed.
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 5 | 1 page per 5 seconds |
bingbot
| Rule | Path |
|---|---|
| Disallow | /ecommercedev/ |
| Disallow | /ecommerce/dwgadmin12/ |
| Disallow | /ecommerce/pc/search.asp |
| Disallow | /ecommerce/pc/showsearchresults.asp |
| Disallow | /v5store/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
msnbot
| Rule | Path |
|---|---|
| Disallow | /ecommercedev/ |
| Disallow | /ecommerce/dwgadmin12/ |
| Disallow | /ecommerce/pc/search.asp |
| Disallow | /ecommerce/pc/showsearchresults.asp |
| Disallow | /v5store/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
slurp
| Rule | Path |
|---|---|
| Disallow | /ecommercedev/ |
| Disallow | /ecommerce/dwgadmin12/ |
| Disallow | /ecommerce/pc/search.asp |
| Disallow | /ecommerce/pc/showsearchresults.asp |
| Disallow | /v5store/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
yandexbot
| Rule | Path |
|---|---|
| Disallow | /ecommercedev/ |
| Disallow | /ecommerce/dwgadmin12/ |
| Disallow | /ecommerce/pc/search.asp |
| Disallow | /ecommerce/pc/showsearchresults.asp |
| Disallow | /v5store/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Comments