Log In Sign Up

maratoneta.it
robots.txt

Robots Exclusion Standard data for maratoneta.it

Resource Scan

Scan Details

Site Domain maratoneta.it
Base Domain maratoneta.it
Scan Status Ok
Last Scan2026-01-23T08:43:21+00:00
Next Scan 2026-01-30T08:43:21+00:00

Last Scan

Scanned2026-01-23T08:43:21+00:00
URL https://maratoneta.it/robots.txt
Domain IPs 2001:4b78:2100:a::8, 95.174.9.148
Response IP 95.174.9.148
Found Yes
Hash 31de927789dda1f17ab27fbfbb5ef8fe5647ff759c72f201ea6d3ad5a9502b27
SimHash 1a1e5d5bafce

Groups

*

Rule Path
Disallow /mildb
Back to top

Comments

  • Small robots.txt
  • More information about this file can be found at
  • <a href="http://www.robotstxt.org/">http://www.robotstxt.org/</a>
  • In case your drupal site is in a subdirectory of your web root (e.g. /drupal)
  • add the name of this directory before the / (slash) below
  • example: Disallow: /drupal/aggregator
  • to stop a polite robot indexing an example dir
  • add a line like: user-agent: polite-bot
  • and: Disallow: /example-dir/
  • Paths (clean URLs)
  • activate crawl if bots are using too much traffic
  • Crawl-Delay: 10
Back to top