Log In Sign Up

sanego.de
robots.txt

Robots Exclusion Standard data for sanego.de

Resource Scan

Scan Details

Site Domain sanego.de
Base Domain sanego.de
Scan Status Ok
Last Scan2024-06-12T10:40:38+00:00
Next Scan 2024-06-19T10:40:38+00:00

Last Scan

Scanned2024-06-12T10:40:38+00:00
URL https://sanego.de/robots.txt
Redirect https://www.sanego.de:443/robots.txt
Redirect Domain www.sanego.de
Redirect Base sanego.de
Domain IPs 3.65.32.70, 52.29.172.52
Redirect IPs 3.65.32.70, 52.29.172.52
Response IP 52.29.172.52
Found Yes
Hash 48048bdb71a3880e44c31c6faad0af00927bebbc5f2f491cf2e5f38514d07c7c
SimHash 2b42d3b14791

Groups

trendictionbot
cliqzbot
mauibot

Rule Path
Disallow /

googlebot
googlebot-mobile
googlebot-image
mediapartners-google
adsbot-google
audisto
screaming frog seo spider
bingbot

No rules defined. All paths allowed.

msnbot
yahoo-mmcrawler
slurp
amazonadbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

*

Rule Path
Disallow /ajax/
Disallow /api/
Disallow /rate/
Disallow /spam/
Disallow /static/
Disallow /userimg/
Disallow /userimage/
Disallow /nl
Disallow /Searchterm/
Disallow /meinsiegel
Disallow /Pageimpression/
Allow /img/statistics/

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.sanego.de/sitemap.gz
Back to top

Comments

  • robots.txt for www.sanego.de
  • block bots ignoring Crawl-delay
  • Set full speed
  • Slightly slower
  • content rules
Back to top

Warnings

  • 1 invalid line.
Back to top