heroparcel.com
robots.txt

Robots Exclusion Standard data for heroparcel.com

Resource Scan

Scan Details

Site Domain heroparcel.com
Base Domain heroparcel.com
Scan Status Ok
Last Scan2025-12-02T07:18:18+00:00
Next Scan 2025-12-09T07:18:18+00:00

Last Scan

Scanned2025-12-02T07:18:18+00:00
URL https://heroparcel.com/robots.txt
Redirect https://www.heroparcel.com/robots.txt
Redirect Domain www.heroparcel.com
Redirect Base heroparcel.com
Domain IPs 46.253.116.84
Redirect IPs 46.253.116.84
Response IP 46.253.116.84
Found Yes
Hash ea42993265046acab26282391805e0eba0207e8e25577a6cc0f9ef846f454a93
SimHash 557cdd70fe2f

Groups

claude-web
claudebot
semrushbot
blexbot
ahrefsbot
dotbot

Rule Path
Disallow /

pinterestbot

Rule Path
Allow /
Disallow /shop/category

Other Records

Field Value
crawl-delay 1.0

gptbot
amazonbot

Rule Path
Allow /
Disallow /shop/category
Disallow /cart

Other Records

Field Value
crawl-delay 1.0

*

Rule Path
Allow /
Disallow /cart

Other Records

Field Value
crawl-delay 0.2

Comments

  • Disallowed crawlers
  • Pinterest
  • Amazon (Alexa), OpenAI GPTBot
  • All crawlers