shivanistylehouse.co.uk
robots.txt
Robots Exclusion Standard data for shivanistylehouse.co.uk
Resource Scan
Scan Details
| Site Domain | shivanistylehouse.co.uk |
| Base Domain | shivanistylehouse.co.uk |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Couldn't connect to server. |
| Last Scan | 2024-10-08T19:41:43+00:00 |
| Next Scan | 2024-10-09T19:41:43+00:00 |
Last Successful Scan
| Scanned | 2024-10-01T19:41:10+00:00 |
| URL | https://shivanistylehouse.co.uk/robots.txt |
| Domain IPs | 209.97.143.241 |
| Response IP | 209.97.143.241 |
| Found | Yes |
| Hash | 954a013adec9ebb8d7e19a934d8d0112b8152d1b878186895b953672a999abd6 |
| SimHash | 0c7c95d076e1 |
Groups
adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
facebot
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
apis-google
storebot-google
mediapartners-google
twitterbot
bingbot
bingpreview
msnbot
msnbot-media
slurp
| Rule | Path |
|---|---|
| Disallow | |
| Disallow | /p/product/adminpanel/ |
| Disallow | /products/ |
| Disallow | /p/product/images/ |
mozilla/5.0 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
baiduspider
yandexbot
mj12bot
ahrefsbot
dotbot
dotbot
dotbot
semrushbot
semrushbot-sa
petalbot
aspiegelbot
blexbot
blexbot/1.0
megaindex.ru
mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)
mozilla/5.0 (compatible; blexbot/1.0; +http://webmeup-crawler.com/)
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | / |
| Disallow | /p/product/adminpanel/ |
| Disallow | /products/ |
| Disallow | /p/product/images/ |
| Disallow | /*.htm |
| Disallow | /*.shtml |
Other Records
| Field | Value |
|---|---|
| sitemap | https://shivanistylehouse.co.uk/sitemap.xml |
| sitemap | https://shivanistylehouse.co.uk/ukproduct-sitemap.xml |
| sitemap | https://shivanistylehouse.co.uk/product-sitemap.xml |
Comments