dd-stroi.ru
robots.txt
Robots Exclusion Standard data for dd-stroi.ru
Resource Scan
Scan Details
| Site Domain | dd-stroi.ru |
| Base Domain | dd-stroi.ru |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-10-06T02:18:30+00:00 |
| Next Scan | 2024-10-13T02:18:30+00:00 |
Last Successful Scan
| Scanned | 2024-07-27T02:15:19+00:00 |
| URL | https://dd-stroi.ru/robots.txt |
| Domain IPs | 37.140.194.102 |
| Response IP | 37.140.194.102 |
| Found | Yes |
| Hash | a78f87adc7d6304162cdea2d7598f3d24137647347fa0b93819117f42330c032 |
| SimHash | 1714c5424331 |
Groups
linguee
ccbot
proximic
blexbot
moget
ichiro
naverbot
yeti
baiduspider
baiduspider-video
baiduspider-image
nutch spider/nutch-1.0-spb
speedy
turnitinbot
npbot-1/2.0
npbot
larbin_2.6.3
sogou spider
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
solomonobot
ahrefsbot
mozilla/5.0 (compatible; ahrefsbot/2.0; +http://ahrefs.com/robot/)
psbot/0.1 (+http://www.picsearch.com/bot.html)
ezooms
mozilla/5.0 (compatible; ezooms/1.0; ezooms.bot@gmail.com)
mj12bot/v1.4.0
mj12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+
discobot
magpie-crawler/1.1 (u; linux amd64; en-gb; +http://www.brandwatch.net)
youdaobot
lcc
semrushbot
semrushbot-sa
serpstatbot
yacybot
| Product | Comment |
|---|---|
| sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm | 07) |
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /*? |
| Disallow | /admin |
| Disallow | /search |
| Disallow | /templates |
| Disallow | /hostcmsfiles |
| Disallow | /captcha.php |
| Disallow | /403 |
| Disallow | /404 |
| Allow | /hostcmsfiles/css/* |
| Allow | /hostcmsfiles/js/* |
| Allow | /upload/* |
| Disallow | /showbanner |
| Disallow | *.swf$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://dd-stroi.ru/sitemap/ |
Warnings
- `host` is not a known field.