media.pennlive.com
robots.txt

Robots Exclusion Standard data for media.pennlive.com

Resource Scan

Scan Details

Site Domain media.pennlive.com
Base Domain pennlive.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-02-02T02:52:37+00:00
Next Scan 2026-04-03T02:52:37+00:00

Last Successful Scan

Scanned2025-11-10T03:13:48+00:00
URL https://media.pennlive.com/robots.txt
Domain IPs 151.101.131.52, 151.101.195.52, 151.101.3.52, 151.101.67.52
Response IP 146.75.47.52
Found Yes
Hash aaeaa0233a181213e57c6136f8182777fa5db3624579ff7e26a6ef6b8294409c
SimHash 42335945d5fe

Groups

bingbot

Rule Path
Disallow /test/
Disallow /preview/
Disallow /staging/
Disallow /clients/
Disallow /auctions/
Disallow /cgi-bin/
Disallow /printer/
Disallow /*/print.html*
Disallow /*/print.ssf*
Disallow /sponsor-content-n/
Disallow /landingpages/
Disallow /oil-spill-reports/
Disallow /premium/
Disallow /contests/scrape/
Disallow /celebrity-news/
Disallow /comics-kingdom/
Disallow /puzzles-games/
Disallow /wedding-stories/
Disallow /mattress/
Disallow /prudential/
Disallow /mid-penn-awnings/
Disallow /us-politics/
Disallow /living/dash/
Disallow /letters/
Disallow /silive_sandbox_blog_-_azywusko
Disallow /toprailapp/
Disallow /*mt-preview-*

googlebot

Rule Path
Disallow /test/
Disallow /preview/
Disallow /staging/
Disallow /clients/
Disallow /auctions/
Disallow /cgi-bin/
Disallow /printer/
Disallow /*/print.html*
Disallow /*/print.ssf*
Disallow /sponsor-content-n/
Disallow /landingpages/
Disallow /oil-spill-reports/
Disallow /premium/
Disallow /contests/scrape/
Disallow /celebrity-news/
Disallow /comics-kingdom/
Disallow /puzzles-games/
Disallow /wedding-stories/
Disallow /mattress/
Disallow /prudential/
Disallow /mid-penn-awnings/
Disallow /us-politics/
Disallow /living/dash/
Disallow /letters/
Disallow /puzzles-palace/
Disallow /silive_sandbox_blog_-_azywusko
Disallow /toprailapp/
Disallow /*mt-preview-*

proximic

Rule Path
Disallow /test/

slurp

Rule Path
Disallow /test/
Disallow /preview/
Disallow /staging/
Disallow /clients/
Disallow /auctions/
Disallow /cgi-bin/
Disallow /printer/
Disallow /*/print.html*
Disallow /*/print.ssf*
Disallow /sponsor-content-n/
Disallow /landingpages/
Disallow /oil-spill-reports/
Disallow /premium/
Disallow /contests/scrape/
Disallow /celebrity-news/
Disallow /comics-kingdom/
Disallow /puzzles-games/
Disallow /wedding-stories/
Disallow /mattress/
Disallow /prudential/
Disallow /mid-penn-awnings/
Disallow /us-politics/
Disallow /living/dash/
Disallow /letters/
Disallow /puzzles-palace/
Disallow /toprailapp/
Disallow /*mt-preview-*

twitterbot

Rule Path
Disallow /test/
Disallow /preview/
Disallow /staging/
Disallow /clients/
Disallow /auctions/
Disallow /cgi-bin/
Disallow /printer/
Disallow /*/print.html*
Disallow /*/print.ssf*
Disallow /sponsor-content-n/
Disallow /landingpages/
Disallow /oil-spill-reports/
Disallow /premium/
Disallow /contests/scrape/
Disallow /celebrity-news/
Disallow /comics-kingdom/
Disallow /puzzles-games/
Disallow /wedding-stories/
Disallow /mattress/
Disallow /prudential/
Disallow /mid-penn-awnings/
Disallow /us-politics/
Disallow /living/dash/
Disallow /letters/
Disallow /puzzles-palace/
Disallow /toprailapp/
Disallow /*mt-preview-*

Other Records

Field Value
sitemap http://www.pennlive.com/sitemap-blog.xml
sitemap http://www.pennlive.com/sitemap.xml
sitemap https://www.pennlive.com/sitemap-secure-blog.xml
sitemap https://www.pennlive.com/sitemap-secure-www.xml