cosmopolitan.com
robots.txt
Robots Exclusion Standard data for cosmopolitan.com
Resource Scan
Scan Details
Site Domain | cosmopolitan.com |
Base Domain | cosmopolitan.com |
Scan Status | Ok |
Last Scan | 2024-10-30T09:31:20+00:00 |
Next Scan | 2024-11-06T09:31:20+00:00 |
Last Scan
Scanned | 2024-10-30T09:31:20+00:00 |
URL | https://cosmopolitan.com/robots.txt |
Domain IPs | 151.101.0.155, 151.101.128.155, 151.101.192.155, 151.101.64.155 |
Response IP | 151.101.192.155 |
Found | Yes |
Hash | ef7a4b118ca43b5dc08e0bfe39611c3c243c693a72521e6c482a5ca545c73209 |
SimHash | 39229f6405f3 |
Groups
oai-searchbot
Rule | Path |
---|---|
Disallow | /au/ |
Disallow | /cn/ |
Disallow | /dk/ |
Disallow | /en/ |
Disallow | /es/ |
Disallow | /in/ |
Disallow | /it/ |
Disallow | /jp/ |
Disallow | /ng/ |
Disallow | /nl/ |
Disallow | /no/ |
Disallow | /se/ |
Disallow | /tw/ |
Disallow | /ua/ |
Disallow | /uk/ |
chatgpt-user
Rule | Path |
---|---|
Disallow | /au/ |
Disallow | /cn/ |
Disallow | /dk/ |
Disallow | /en/ |
Disallow | /es/ |
Disallow | /in/ |
Disallow | /it/ |
Disallow | /jp/ |
Disallow | /ng/ |
Disallow | /nl/ |
Disallow | /no/ |
Disallow | /se/ |
Disallow | /tw/ |
Disallow | /ua/ |
Disallow | /uk/ |
gptbot
Rule | Path |
---|---|
Disallow | /au/ |
Disallow | /cn/ |
Disallow | /dk/ |
Disallow | /en/ |
Disallow | /es/ |
Disallow | /in/ |
Disallow | /it/ |
Disallow | /jp/ |
Disallow | /ng/ |
Disallow | /nl/ |
Disallow | /no/ |
Disallow | /se/ |
Disallow | /tw/ |
Disallow | /ua/ |
Disallow | /uk/ |
*
Rule | Path |
---|---|
Disallow | /_p/ |
Disallow | /afflink/ |
Disallow | /ajax/ |
Disallow | /auth/ |
Disallow | /au/ajax/ |
Disallow | /cn/ajax/ |
Disallow | /dk/ajax/ |
Disallow | /en/ajax/ |
Disallow | /es/ajax/ |
Disallow | /in/ajax/ |
Disallow | /it/ajax/ |
Disallow | /jp/ajax/ |
Disallow | /ng/ajax/ |
Disallow | /nl/ajax/ |
Disallow | /no/ajax/ |
Disallow | /se/ajax/ |
Disallow | /tw/ajax/ |
Disallow | /ua/ajax/ |
Disallow | /uk/ajax/ |
Disallow | /landing-feed/ |
Disallow | /oauth/ |
Disallow | /preview/ |
Disallow | /au/preview/ |
Disallow | /cn/preview/ |
Disallow | /dk/preview/ |
Disallow | /en/preview/ |
Disallow | /es/preview/ |
Disallow | /in/preview/ |
Disallow | /it/preview/ |
Disallow | /jp/preview/ |
Disallow | /ng/preview/ |
Disallow | /nl/preview/ |
Disallow | /no/preview/ |
Disallow | /se/preview/ |
Disallow | /tw/preview/ |
Disallow | /ua/preview/ |
Disallow | /uk/preview/ |
Disallow | /search |
Disallow | /transporter/ |
Allow | /ajax/content-product/ |
Allow | /au/ajax/content-product/ |
Allow | /cn/ajax/content-product/ |
Allow | /dk/ajax/content-product/ |
Allow | /en/ajax/content-product/ |
Allow | /es/ajax/content-product/ |
Allow | /in/ajax/content-product/ |
Allow | /it/ajax/content-product/ |
Allow | /jp/ajax/content-product/ |
Allow | /ng/ajax/content-product/ |
Allow | /nl/ajax/content-product/ |
Allow | /no/ajax/content-product/ |
Allow | /se/ajax/content-product/ |
Allow | /tw/ajax/content-product/ |
Allow | /ua/ajax/content-product/ |
Allow | /uk/ajax/content-product/ |
Allow | /ajax/contentmedia/ |
Allow | /au/ajax/contentmedia/ |
Allow | /cn/ajax/contentmedia/ |
Allow | /dk/ajax/contentmedia/ |
Allow | /en/ajax/contentmedia/ |
Allow | /es/ajax/contentmedia/ |
Allow | /in/ajax/contentmedia/ |
Allow | /it/ajax/contentmedia/ |
Allow | /jp/ajax/contentmedia/ |
Allow | /ng/ajax/contentmedia/ |
Allow | /nl/ajax/contentmedia/ |
Allow | /no/ajax/contentmedia/ |
Allow | /se/ajax/contentmedia/ |
Allow | /tw/ajax/contentmedia/ |
Allow | /ua/ajax/contentmedia/ |
Allow | /uk/ajax/contentmedia/ |
Other Records
Field | Value |
---|---|
sitemap | https://www.cosmopolitan.com/sitemap_index.xml |
sitemap | https://www.cosmopolitan.com/sitemap_google_news.xml |
sitemap | https://www.cosmopolitan.com/uk/sitemap_index.xml |
sitemap | https://www.cosmopolitan.com/uk/sitemap_google_news.xml |