paulwesterberg.com
robots.txt

Robots Exclusion Standard data for paulwesterberg.com

Resource Scan

Scan Details

Site Domain paulwesterberg.com
Base Domain paulwesterberg.com
Scan Status Ok
Last Scan2025-04-15T00:11:42+00:00
Next Scan 2025-05-15T00:11:42+00:00

Last Scan

Scanned2025-04-15T00:11:42+00:00
URL https://paulwesterberg.com/robots.txt
Redirect https://rakhoivx.cc/robots.txt
Redirect Domain rakhoivx.cc
Redirect Base rakhoivx.cc
Domain IPs 104.21.64.48, 172.67.176.12, 2606:4700:3030::ac43:b00c, 2606:4700:3031::6815:4030
Redirect IPs 104.21.23.120, 172.67.210.215, 2606:4700:3033::6815:1778, 2606:4700:3037::ac43:d2d7
Response IP 104.21.23.120
Found Yes
Hash f2e9cd1441331441b89d85c7b8288fb0240a0195d0ab83595217eca8991cc9cb
SimHash 6bc9dc04c3b2

Groups

*

Rule Path
Allow /
Disallow /wp-admin/
Disallow /*/trackback
Disallow /img/
Disallow /tag/
Disallow /feed
Disallow /*/feed
Disallow /?s=*
Disallow /?link=*
Disallow /attachment/
Disallow /author/
Disallow /page/*

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
sitemap https://rakhoivx.cc/sitemap.xml

Comments

  • Allow Facebook scraper