semesterboenden.com
robots.txt

Robots Exclusion Standard data for semesterboenden.com

Resource Scan

Scan Details

Site Domain semesterboenden.com
Base Domain semesterboenden.com
Scan Status Ok
Last Scan2025-10-31T14:10:53+00:00
Next Scan 2025-11-07T14:10:53+00:00

Last Scan

Scanned2025-10-31T14:10:53+00:00
URL https://semesterboenden.com/robots.txt
Redirect https://www.semesterboenden.com/robots.txt
Redirect Domain www.semesterboenden.com
Redirect Base semesterboenden.com
Domain IPs 34.163.220.84
Redirect IPs 34.163.220.84
Response IP 34.163.220.84
Found Yes
Hash 6c46ae492fb2423a3bf94761f2c5d89025b103222fd7bb7202ec46b7042a2888
SimHash d25a8212ecb8

Groups

*

Rule Path
Disallow /search?
Disallow /api/
Disallow /redirect
Disallow /next/

bubing

Rule Path
Disallow /

genieo

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value Comment
crawl-delay 4.5 specifies a 2 second timeout