hosiwien.at
robots.txt

Robots Exclusion Standard data for hosiwien.at

Resource Scan

Scan Details

Site Domain hosiwien.at
Base Domain hosiwien.at
Scan Status Ok
Last Scan2024-09-25T14:50:22+00:00
Next Scan 2024-10-25T14:50:22+00:00

Last Scan

Scanned2024-09-25T14:50:22+00:00
URL https://hosiwien.at/robots.txt
Redirect https://www.hosiwien.at/robots.txt
Redirect Domain www.hosiwien.at
Redirect Base hosiwien.at
Domain IPs 83.169.1.42
Redirect IPs 83.169.1.42
Response IP 83.169.1.42
Found Yes
Hash 5ebe4d7cd56224bf02d7dd2a72382801c601e79a763da45d6e91e334ae67b92a
SimHash ce0ac42195f1

Groups

*

Rule Path
Disallow /wp-json/
Disallow /wsm/
Disallow /img/
Disallow /cgi-local/
Disallow /forms/
Disallow /stats/
Disallow /bannerimg/
Disallow /download/

googlebot

Rule Path
Allow /img/

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

amazonbot

Rule Path
Disallow *

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /

Comments

  • Global rules
  • -----------------
  • Allow Googlebot to index imgs
  • -----------------
  • Slow down MSN/Bing/Yahoo
  • Disallow Amazonbot
  • Ban bots that don't benefit us.
  • --------------------------------