penews.com
robots.txt
Robots Exclusion Standard data for penews.com
Resource Scan
Scan Details
| Site Domain | penews.com |
| Base Domain | penews.com |
| Scan Status | Ok |
| Last Scan | 2024-11-12T02:43:06+00:00 |
| Next Scan | 2024-11-19T02:43:06+00:00 |
Last Scan
| Scanned | 2024-11-12T02:43:06+00:00 |
| URL | https://www.penews.com/robots.txt |
| Domain IPs | 13.35.238.107, 13.35.238.41, 13.35.238.95, 13.35.238.96 |
| Response IP | 13.35.238.95 |
| Found | Yes |
| Hash | 8e10c2b88525171a8b386a69c14b5e99e1dffd8149416e2b77a9837296762ad0 |
| SimHash | 3809c047c5d0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /profile/* |
| Disallow | /search/* |
| Disallow | /client* |
| Disallow | /auth/* |
| Disallow | /user/* |
| Disallow | /newsletters/svc/* |
| Disallow | /_next/* |
| Disallow | /asset/* |
| Disallow | /forms/* |
| Disallow | /follow/* |
| Disallow | /api/* |
| Disallow | /email_most-read/* |
| Disallow | /cookies* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.penews.com/sitemap.xml |
| sitemap | https://www.penews.com/pen_google_news.xml |
| sitemap | https://www.penews.com/sitemaps/web/pen/en/sitemap_pen_en_index.xml |