Log In Sign Up

fnal.gov
robots.txt

Robots Exclusion Standard data for fnal.gov

Resource Scan

Scan Details

Site Domain fnal.gov
Base Domain fnal.gov
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-02-05T05:13:40+00:00
Next Scan 2026-02-19T05:13:40+00:00

Last Successful Scan

Scanned2026-01-14T05:10:24+00:00
URL https://fnal.gov/robots.txt
Domain IPs 131.225.105.119, 2620:6a:0:105:f0:0:105:119
Response IP 131.225.105.119
Found Yes
Hash 86bcaba6878a237d7e815eff4868355d32b0bb9e1b7ae279285e19820d817e7e
SimHash 3e037f06c814

Groups

*

Rule Path
Disallow /logs/
Disallow /reports/WWWlogs/
Disallow /docs/WWWlogs/
Disallow /reports/
Disallow /faw/events/
Disallow /faw/atwork/
Disallow /faw/fermiworks_dev/
Disallow /cd/guidelines/
Disallow /cd/webgroup/internal/
Disallow /docs/UNIX/unix_at_fermilab/htmldoc/rev093096/
Disallow /docs/UNIX/unix_at_fermilab/htmldoc/rev1997b/
Disallow /docs/UNIX/unix_at_fermilab/htmldoc/rev1997test1/
Disallow /docs/UNIX/unix_at_fermilab/htmldoc/rev97indiv/
Disallow /docs/UNIX/product_methodology/html/rev1_1a-orig/
Disallow /docs/obsolete
Disallow /docs/working-groups/fpcltf/Pkg/SoftRelTools/SoftRelTools/
Disallow /~
Disallow /cgi-bin/
Disallow /htbin/
Disallow /pub/today/aprilfoolsaccelupdate.html
Disallow /pub/today/aprilfools.html
Disallow /pub/today/aprildefinition.html
Disallow /projects/history_old/
Disallow /directorate/Longrange/LRPC/auth/.htpasswd
Disallow /directorate/ELM/auth/.htpasswd
Disallow /ecology/.auth/.htpasswd
Disallow /tools/hypernews/hnpeople/.htpasswd
Disallow /tools/hypernews/hnpeople/.htpasswd-old
Disallow /docs/products/xemacs/
Disallow /orgs/barnclub/
Disallow /orgs/chess/
Disallow /orgs/choir/
Disallow /orgs/far/
Disallow /orgs/radioclub/
Disallow /orgs/writers/

slurp

Rule Path
Disallow /

msnbot

Rule Path
Disallow /orgs/gsa/
Back to top

Comments

  • disallow xemacs because Slurp (inktomi) accesses to xemacs/v21_1
  • coincide with malloc errors on the web server - LMengel 9/18/2000
  • disallow orgs recreation area dirs (per Jean Guyer, Bruce Chrisman)
  • Laura Mengel 2007/08/02
  • disallow Slurp (inktomi) accesses to whole server
  • it is accessing our site more than the advertised once/min,
  • sometimes many times per second. This coincides with malloc
  • errors on the web server - LMengel 9/24/2000
  • disallow msnbot accesses to the GSA areas
  • FirstGov uses MSN as their customized search engine
  • Image searches at FirstGov were showing Fermi's GSA pictures
  • instead of gsa.gov pictures.
  • Fermilab's GSA officers decided it was best to just block
  • the msnbot from accessing their area. TDawson 11/18/2006
Back to top