radioviainternet.be
robots.txt

Robots Exclusion Standard data for radioviainternet.be

Resource Scan

Scan Details

Site Domain radioviainternet.be
Base Domain radioviainternet.be
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-05-09T06:21:32+00:00
Next Scan 2025-08-07T06:21:32+00:00

Last Successful Scan

Scanned2025-01-10T06:19:44+00:00
URL https://radioviainternet.be/robots.txt
Redirect https://www.radioviainternet.be/robots.txt
Redirect Domain www.radioviainternet.be
Redirect Base radioviainternet.be
Domain IPs 104.21.10.25, 172.67.162.36, 2606:4700:3032::ac43:a224, 2606:4700:3033::6815:a19
Redirect IPs 104.21.10.25, 172.67.162.36, 2606:4700:3032::ac43:a224, 2606:4700:3033::6815:a19
Response IP 104.21.10.25
Found Yes
Hash 98e6c755da6fac5256cd726088cec0267b14556fe45f9a599ffa9cbd6822facd
SimHash b001d8108031

Groups

*

Rule Path
Disallow
Disallow /i/
Disallow /embed/

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2