Log In Sign Up

mit.bme.hu
robots.txt

Robots Exclusion Standard data for mit.bme.hu

Resource Scan

Scan Details

Site Domain mit.bme.hu
Base Domain bme.hu
Scan Status Ok
Last Scan2025-08-25T07:02:01+00:00
Next Scan 2025-09-24T07:02:01+00:00

Last Scan

Scanned2025-08-25T07:02:01+00:00
URL http://mit.bme.hu/robots.txt
Redirect https://www.mit.bme.hu/robots.txt
Redirect Domain www.mit.bme.hu
Redirect Base bme.hu
Domain IPs 152.66.252.1
Redirect IPs 152.66.252.13
Response IP 152.66.252.13
Found Yes
Hash 3fb5f59c9d04eb07f27b05f701843fc3cc7c7fdfc5fb39a5e89b0889a5a8a789
SimHash 5a165d5bafde

Groups

*

Rule Path
Disallow /comment/reply
Disallow /node/add
Disallow /user
Disallow /search
Disallow /system/files/oktatas/targyak/vedett

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Small robots.txt
  • More information about this file can be found at
  • <a href="http://www.robotstxt.org/">http://www.robotstxt.org/</a>
  • In case your drupal site is in a subdirectory of your web root (e.g.
  • /drupal)
  • add the name of this directory before the / (slash) below
  • example: Disallow: /drupal/aggregator
  • to stop a polite robot indexing an example dir
  • add a line like: user-agent: polite-bot
  • and: Disallow: /example-dir/
  • Paths (clean URLs)
Back to top

Warnings

  • 1 invalid line.
Back to top