thepioneerwoman.com
robots.txt

Robots Exclusion Standard data for thepioneerwoman.com

Resource Scan

Scan Details

Site Domain thepioneerwoman.com
Base Domain thepioneerwoman.com
Scan Status Ok
Last Scan2024-11-09T17:59:35+00:00
Next Scan 2024-11-16T17:59:35+00:00

Last Scan

Scanned2024-11-09T17:59:35+00:00
URL https://thepioneerwoman.com/robots.txt
Domain IPs 151.101.0.155, 151.101.128.155, 151.101.192.155, 151.101.64.155
Response IP 151.101.128.155
Found Yes
Hash 541714a0d1f92641a3e29f81b2fcf3c409cb5bd5733b27f7304085a8673616da
SimHash f92a5fe581ff

Groups

mediapartners-google

Rule Path
Disallow

facebot

Rule Path
Disallow

twitterbot

Rule Path
Disallow

ia_archiver

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

*

Rule Path
Disallow /_p/
Disallow /afflink/
Disallow /ajax/
Disallow /auth/
Disallow /au/ajax/
Disallow /cn/ajax/
Disallow /dk/ajax/
Disallow /en/ajax/
Disallow /es/ajax/
Disallow /in/ajax/
Disallow /it/ajax/
Disallow /jp/ajax/
Disallow /ng/ajax/
Disallow /nl/ajax/
Disallow /no/ajax/
Disallow /se/ajax/
Disallow /tw/ajax/
Disallow /ua/ajax/
Disallow /uk/ajax/
Disallow /landing-feed/
Disallow /oauth/
Disallow /preview/
Disallow /au/preview/
Disallow /cn/preview/
Disallow /dk/preview/
Disallow /en/preview/
Disallow /es/preview/
Disallow /in/preview/
Disallow /it/preview/
Disallow /jp/preview/
Disallow /ng/preview/
Disallow /nl/preview/
Disallow /no/preview/
Disallow /se/preview/
Disallow /tw/preview/
Disallow /ua/preview/
Disallow /uk/preview/
Disallow /search
Disallow /transporter/
Allow /ajax/content-product/
Allow /au/ajax/content-product/
Allow /cn/ajax/content-product/
Allow /dk/ajax/content-product/
Allow /en/ajax/content-product/
Allow /es/ajax/content-product/
Allow /in/ajax/content-product/
Allow /it/ajax/content-product/
Allow /jp/ajax/content-product/
Allow /ng/ajax/content-product/
Allow /nl/ajax/content-product/
Allow /no/ajax/content-product/
Allow /se/ajax/content-product/
Allow /tw/ajax/content-product/
Allow /ua/ajax/content-product/
Allow /uk/ajax/content-product/
Allow /ajax/contentmedia/
Allow /au/ajax/contentmedia/
Allow /cn/ajax/contentmedia/
Allow /dk/ajax/contentmedia/
Allow /en/ajax/contentmedia/
Allow /es/ajax/contentmedia/
Allow /in/ajax/contentmedia/
Allow /it/ajax/contentmedia/
Allow /jp/ajax/contentmedia/
Allow /ng/ajax/contentmedia/
Allow /nl/ajax/contentmedia/
Allow /no/ajax/contentmedia/
Allow /se/ajax/contentmedia/
Allow /tw/ajax/contentmedia/
Allow /ua/ajax/contentmedia/
Allow /uk/ajax/contentmedia/

Other Records

Field Value
sitemap https://www.thepioneerwoman.com/sitemap_index.xml
sitemap https://www.thepioneerwoman.com/sitemap_google_news.xml