Log In Sign Up

gnome-db.org
robots.txt

Robots Exclusion Standard data for gnome-db.org

Resource Scan

Scan Details

Site Domain gnome-db.org
Base Domain gnome-db.org
Scan Status Ok
Last Scan2025-07-25T16:17:36+00:00
Next Scan 2025-08-24T16:17:36+00:00

Last Scan

Scanned2025-07-25T16:17:36+00:00
URL https://gnome-db.org/robots.txt
Redirect https://www.gnome-db.org/robots.txt
Redirect Domain www.gnome-db.org
Redirect Base gnome-db.org
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Redirect IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.32.1
Found Yes
Hash 2fbf567f7f06d57e997e3339ae32be39d58aa1a2dae480001f10d194d622e093
SimHash 2c7a5f024f35

Groups

dotbot

Rule Path
Disallow /

*

Rule Path
Disallow /action/

Other Records

Field Value
crawl-delay 20
Back to top

Comments

  • if you want to add own robot rules, do it BEFORE the final rule matching *
  • This has to match script url + cfg.url_prefix_action - it
  • saves lots of search engine load and traffic by disallowing crawlers
  • to request action related URLs.
  • NOTE - in order to make this have any effect, you have to set
  • url_prefix_action to "action", cf. HelpOnConfiguration
Back to top