mfd.ru
robots.txt
Robots Exclusion Standard data for mfd.ru
Resource Scan
Scan Details
| Site Domain | mfd.ru |
| Base Domain | mfd.ru |
| Scan Status | Ok |
| Last Scan | 2024-11-16T10:37:26+00:00 |
| Next Scan | 2024-11-23T10:37:26+00:00 |
Last Scan
| Scanned | 2024-11-16T10:37:26+00:00 |
| URL | https://mfd.ru/robots.txt |
| Domain IPs | 185.129.103.70 |
| Response IP | 185.129.103.70 |
| Found | Yes |
| Hash | 35c6e04ac4475da7b63c8e71f75b03871256ff5a72ad6cbb3b5a6ceff9a170d9 |
| SimHash | cfd1ce6987f1 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /Admin |
| Disallow | /blogs/search |
| Disallow | /Blogs/Search |
| Disallow | /calendar/earnings/search |
| Disallow | /Calendar/Earnings/Search |
| Disallow | /forum/forecasters |
| Disallow | /Forum/Forecasters |
| Disallow | /forum/post/create |
| Disallow | /Forum/Post/Create |
| Disallow | /forum/post/update |
| Disallow | /Forum/Post/Update |
| Disallow | /forum/poster |
| Disallow | /Forum/Poster |
| Disallow | /forum/search |
| Disallow | /Forum/Search |
| Disallow | /forum/thread/new |
| Disallow | /Forum/Thread/New |
| Disallow | /forum/users |
| Disallow | /Forum/Users |
| Disallow | /marketdata/search |
| Disallow | /MarketData/Search |
| Disallow | /marketdata/tables |
| Disallow | /MarketData/Tables |
| Disallow | /news/search |
| Disallow | /News/Search |
| Disallow | /search |
| Disallow | /Search |
| Disallow | /user |
| Disallow | /User |
| Disallow | /*.css$ |
| Disallow | /*.js$ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://mfd.ru/sitemap.ashx |
Warnings
- `host` is not a known field.
- `request-rate` is not a known field.