Log In Sign Up

benzelbusch.com
robots.txt

Robots Exclusion Standard data for benzelbusch.com

Resource Scan

Scan Details

Site Domain benzelbusch.com
Base Domain benzelbusch.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-15T18:43:06+00:00
Next Scan 2025-12-14T18:43:06+00:00

Last Successful Scan

Scanned2023-11-03T14:13:10+00:00
URL https://benzelbusch.com/robots.txt
Redirect https://www.benzelbusch.com/robots.txt
Redirect Domain www.benzelbusch.com
Redirect Base benzelbusch.com
Domain IPs 151.101.1.193, 151.101.129.193, 151.101.193.193, 151.101.65.193
Redirect IPs 151.101.1.193, 151.101.129.193, 151.101.193.193, 151.101.65.193, 2a04:4e42:200::449, 2a04:4e42:400::449, 2a04:4e42:600::449, 2a04:4e42::449
Response IP 199.232.45.193
Found Yes
Hash 95ae7fe3866455fc2c9efe5d4e6d54dde56e9ef7e84ce62cff1ce800a7adf941
SimHash 3994bd51c760

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /modules/*.css$
Allow /modules/*.css?
Allow /modules/*.js$
Allow /modules/*.js?
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /modules/*.svg
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Allow /themes/*.css$
Allow /themes/*.css?
Allow /themes/*.js$
Allow /themes/*.js?
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Allow /themes/*.svg
Disallow /core/
Disallow /modules/
Disallow /profiles/
Disallow /themes/
Disallow /README.txt
Disallow /web.config
Disallow /update.php
Disallow /INSTALL.txt
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password
Disallow /index.php/user/register
Disallow /index.php/user/login
Disallow /index.php/user/logout

Other Records

Field Value
crawl-delay 10

googlebot-image

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.benzelbusch.com/sitemap.xml
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Invite Google images bot
Back to top