Log In Sign Up

rogoblen.ro
robots.txt

Robots Exclusion Standard data for rogoblen.ro

Resource Scan

Scan Details

Site Domain rogoblen.ro
Base Domain rogoblen.ro
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-30T13:31:56+00:00
Next Scan 2025-11-06T13:31:56+00:00

Last Successful Scan

Scanned2025-09-28T22:18:07+00:00
URL https://rogoblen.ro/robots.txt
Redirect https://www.rogoblen.ro/robots.txt
Redirect Domain www.rogoblen.ro
Redirect Base rogoblen.ro
Domain IPs 138.201.122.57
Redirect IPs 138.201.122.57
Response IP 138.201.122.57
Found Yes
Hash 5276085f3bb3911d2944c6abb0bf033666979c6077ff23c858712701e46d50e1
SimHash 944fd632f9f8

Groups

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

qwantify

Rule Path
Disallow /
Back to top

Comments

  • Block CazoodleBot as it does not present correct accept content headers
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block dotbot as it cannot parse base urls properly
Back to top