walterknoll.de
robots.txt

Robots Exclusion Standard data for walterknoll.de

Resource Scan

Scan Details

Site Domain walterknoll.de
Base Domain walterknoll.de
Scan Status Ok
Last Scan2024-10-21T12:17:49+00:00
Next Scan 2024-11-20T12:17:49+00:00

Last Scan

Scanned2024-10-21T12:17:49+00:00
URL https://www.walterknoll.de/robots.txt
Domain IPs 76.76.21.123, 76.76.21.164
Response IP 76.76.21.164
Found Yes
Hash ad118c11e3db4540df0bc93d1b36fe9762e54dbeed2c118bb723ed15d03eb4fd
SimHash 6d441b13c5d1

Groups

*

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

adsbot-google-mobile-apps

Rule Path
Allow /

*

Rule Path
Disallow /*preview

*

Rule Path
Disallow /*banners

Comments

  • Block googlebot from example.com/directory1/... and example.com/directory2/...
  • but allow access to directory2/subdirectory1/...
  • All other directories on the site are allowed by default.
  • Block the entire site from anothercrawler.
  • User-agent: googlebot
  • Disallow: /