news24.com
robots.txt
Robots Exclusion Standard data for news24.com
Resource Scan
Scan Details
Site Domain | news24.com |
Base Domain | news24.com |
Scan Status | Ok |
Last Scan | 2024-09-23T02:18:37+00:00 |
Next Scan | 2024-09-30T02:18:37+00:00 |
Last Scan
Scanned | 2024-09-23T02:18:37+00:00 |
URL | https://news24.com/robots.txt |
Redirect | https://www.news24.com/robots.txt |
Redirect Domain | www.news24.com |
Redirect Base | news24.com |
Domain IPs | 104.17.250.6, 104.17.251.6, 2606:4700::6811:fa06, 2606:4700::6811:fb06 |
Redirect IPs | 104.17.250.6, 104.17.251.6, 2606:4700::6811:fa06, 2606:4700::6811:fb06 |
Response IP | 104.17.251.6 |
Found | Yes |
Hash | 9a6d339efd2bbda3b32486d246c3ca601e1ea1b2f3c9e1d59ab8aa2cc7a1fe69 |
SimHash | 741c5951edb1 |
Groups
*
Rule | Path |
---|---|
Allow | / |
Allow | */cricket/test/ |
Allow | */cricketworldcup2019/Test/ |
Disallow | /toets/ |
Disallow | */toets/* |
Disallow | /test/ |
Disallow | */_test/* |
Disallow | */testpolar/* |
Disallow | */test/* |
Disallow | /xArchive/Archive/Illegal-liquor-export-20010319 |
Disallow | /.well-known/ |
Disallow | /assetlinks.json |
Other Records
Field | Value |
---|---|
sitemap | https://www.news24.com/sitemap |
Warnings
- 2 invalid lines.
Comments