worthepenny.com
robots.txt

Robots Exclusion Standard data for worthepenny.com

Resource Scan

Scan Details

Site Domain worthepenny.com
Base Domain worthepenny.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-05T20:59:11+00:00
Next Scan 2024-08-03T20:59:11+00:00

Last Successful Scan

Scanned2023-07-11T20:13:33+00:00
URL https://worthepenny.com/robots.txt
Redirect https://www.worthepenny.com/robots.txt
Redirect Domain www.worthepenny.com
Redirect Base worthepenny.com
Domain IPs 104.21.89.144, 172.67.160.221, 2606:4700:3031::6815:5990, 2606:4700:3031::ac43:a0dd
Redirect IPs 104.21.89.144, 172.67.160.221, 2606:4700:3031::6815:5990, 2606:4700:3031::ac43:a0dd
Response IP 104.21.89.144
Found Yes
Hash 9b8f2efd981abb712f4bf63b74a5b11796f4a77f581df2ae7a413f2503947164
SimHash ea06d23c97b5

Groups

jooblebot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

mediapartners-google

Rule Path
Allow /

bingpreview
adsbot-google-mobile
bingbot
adidxbot
*

Rule Path
Allow /$
Allow /sitemap.txt$
Allow /sitemap.xml$
Allow /sitemap1.xml$
Allow /military-discounts/$
Allow /student-discounts/$
Allow /select$
Allow /trending$
Allow /theiconic-coupon$
Allow /ads.txt$
Allow /wep/
Allow /wep-img/
Allow /upload/
Allow /archive/
Allow /articles/
Allow /*.html$
Disallow /

Comments

  • worthepenny