Log In Sign Up

hedgehog.markets
robots.txt

Robots Exclusion Standard data for hedgehog.markets

Resource Scan

Scan Details

Site Domain hedgehog.markets
Base Domain hedgehog.markets
Scan Status Ok
Last Scan2024-09-18T06:54:11+00:00
Next Scan 2024-09-25T06:54:11+00:00

Last Scan

Scanned2024-09-18T06:54:11+00:00
URL https://hedgehog.markets/robots.txt
Domain IPs 13.251.96.10, 52.74.166.77
Response IP 13.251.96.10
Found Yes
Hash 151d51ac6250e9f044308d47e48ad26c80ca0e4a588a9bdc62a4cf66da63e615
SimHash 3c101510c9e1

Groups

*

Rule Path
Disallow

twitterbot

Rule Path
Allow /images

facebookexternalhit

Rule Path
Allow /images
Back to top

Comments

  • https://www.robotstxt.org/robotstxt.html
  • Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.
Back to top