penews.com
robots.txt
Robots Exclusion Standard data for penews.com
Resource Scan
Scan Details
| Site Domain | penews.com |
| Base Domain | penews.com |
| Scan Status | Ok |
| Last Scan | 2024-05-27T14:22:33+00:00 |
| Next Scan | 2024-06-03T14:22:33+00:00 |
Last Scan
| Scanned | 2024-05-27T14:22:33+00:00 |
| URL | https://www.penews.com/robots.txt |
| Domain IPs | 52.84.229.124, 52.84.229.51, 52.84.229.58, 52.84.229.8 |
| Response IP | 52.84.229.58 |
| Found | Yes |
| Hash | 9ddfbdf7aa59ad8148da57f89d1e2708ddb983b4bbb920483a2561475abd6230 |
| SimHash | 38098847cd90 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /profile/* |
| Disallow | /search/* |
| Disallow | /client* |
| Disallow | /auth/* |
| Disallow | /user/* |
| Disallow | /newsletters/svc/* |
| Disallow | /_next/* |
| Disallow | /asset/* |
| Disallow | /forms/* |
| Disallow | /follow/* |
| Disallow | /api/* |
| Disallow | /email_most-read/* |
| Disallow | /cookies* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.penews.com/sitemap.xml |
| sitemap | https://www.penews.com/pen_google_news.xml |
| sitemap | https://www.penews.com/sitemaps/web/pen/en/sitemap_pen_en_index.xml |