thewebster.com
robots.txt
Robots Exclusion Standard data for thewebster.com
Resource Scan
Scan Details
| Site Domain | thewebster.com |
| Base Domain | thewebster.com |
| Scan Status | Ok |
| Last Scan | 2025-08-13T11:51:06+00:00 |
| Next Scan | 2025-09-12T11:51:06+00:00 |
Last Scan
| Scanned | 2025-08-13T11:51:06+00:00 |
| URL | https://thewebster.com/robots.txt |
| Domain IPs | 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124 |
| Response IP | 151.101.1.124 |
| Found | Yes |
| Hash | bb40d96b6f4ac182fd47e538865f363651d0b8de432d3456795d73f85b1854a1 |
| SimHash | d405101a4bda |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /shop/checkout* |
| Disallow | /shop/onestepcheckout* |
| Disallow | /shop/customer* |
| Disallow | /shop/customer/account* |
| Disallow | /shop/customer/account/login* |
| Disallow | /shop/guestwishlist* |
| Disallow | /digitalsupport* |
| Disallow | /shop/catalogsearch/advanced* |
| Disallow | /shop/giftregistry* |
| Disallow | /shop/sales/order/history* |
| Disallow | /shop/catalogsearch/ |
| Disallow | /shop/catalog/product_compare* |
| Disallow | /shop/catalog/category/view* |
| Disallow | /shop/catalog/product/view* |
| Disallow | /*?dir* |
| Disallow | /*?dir=desc |
| Disallow | /*?dir=asc |
| Disallow | /*?limit=all |
| Disallow | /?mode |
| Disallow | /shop/aw_blog/block/render/id/* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://thewebster.us/sitemap_index.xml |
| sitemap | https://thewebster.com/media/google_sitemap_category.xml |
| sitemap | https://thewebster.com/media/sitemap_index.xml |
| sitemap | https://thewebster.com/media/google_sitemap_editorial.xml |
| sitemap | https://thewebster.com/media/google_sitemap_cms.xml |
| sitemap | https://thewebster.com/media/google_sitemap_designers.xml |
Comments