thepost.co.za
robots.txt
Robots Exclusion Standard data for thepost.co.za
Resource Scan
Scan Details
| Site Domain | thepost.co.za |
| Base Domain | thepost.co.za |
| Scan Status | Ok |
| Last Scan | 2024-06-09T13:43:36+00:00 |
| Next Scan | 2024-06-16T13:43:36+00:00 |
Last Scan
| Scanned | 2024-06-09T13:43:36+00:00 |
| URL | https://thepost.co.za/robots.txt |
| Domain IPs | 104.21.84.234, 172.67.198.223, 2606:4700:3035::6815:54ea, 2606:4700:3035::ac43:c6df |
| Response IP | 104.21.84.234 |
| Found | Yes |
| Hash | 5fce38735b9f3c5a24d079db6ed19cbfbf5406cf16d5193652af0762c7d1c279 |
| SimHash | c9f61c04fff1 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | *?z= |
| Disallow | *?q= |
| Disallow | */preview/ |
| Disallow | /menu/ |
| Disallow | /menu |
| Disallow | /menu?* |
| Disallow | /print-content* |
| Disallow | /widgets/rss_redirect.php?* |
| Disallow | /index.php?* |
| Disallow | /test$ |
| Disallow | /test1$ |
| Disallow | /test2$ |
| Disallow | /my-notifications |
| Disallow | /my-bookmarks |
| Disallow | /business-report/test$ |
| Disallow | /news/test$ |
| Disallow | /personal-finance/test$ |
| Disallow | /sport/test$ |
| Disallow | /entertainment/test$ |
| Disallow | /lifestyle/test$ |
| Disallow | /travel/test$ |
| Disallow | /motoring/test$ |
| Disallow | /wap.iol.co.za/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.iol.co.za/sitemap.xml |