businessinsider.co.za
robots.txt
Robots Exclusion Standard data for businessinsider.co.za
Resource Scan
Scan Details
Site Domain | businessinsider.co.za |
Base Domain | businessinsider.co.za |
Scan Status | Failed |
Failure Stage | Fetching resource. |
Failure Reason | Server returned a server error. |
Last Scan | 2024-09-04T11:35:21+00:00 |
Next Scan | 2024-12-03T11:35:21+00:00 |
Last Successful Scan
Scanned | 2023-08-12T11:24:57+00:00 |
URL | https://businessinsider.co.za/robots.txt |
Redirect | https://www.businessinsider.co.za/robots.txt |
Redirect Domain | www.businessinsider.co.za |
Redirect Base | businessinsider.co.za |
Domain IPs | 104.25.11.119, 104.25.12.119, 2606:4700:20::6819:b77, 2606:4700:20::6819:c77 |
Redirect IPs | 104.25.11.119, 104.25.12.119, 2606:4700:20::6819:b77, 2606:4700:20::6819:c77 |
Response IP | 104.25.11.119 |
Found | Yes |
Hash | 02458613e55c3a07e107a124b4f5333344b4bcc2f29f65d728daf64ac82b17a5 |
SimHash | 4904c811dfd1 |
Groups
*
Rule | Path |
---|---|
Allow | / |
Allow | */cricket/test/ |
Allow | */cricketworldcup2019/Test/ |
Disallow | /toets/ |
Disallow | */toets/* |
Disallow | /test/ |
Disallow | */_test/* |
Disallow | */testpolar/* |
Disallow | */test/* |
Disallow | /xArchive/Archive/Illegal-liquor-export-20010319 |
Disallow | /.well-known/ |
Disallow | /assetlinks.json |
Other Records
Field | Value |
---|---|
sitemap | https://www.businessinsider.com/sitemap |