Log In Sign Up

ftp.gnu.org
robots.txt

Robots Exclusion Standard data for ftp.gnu.org

Resource Scan

Scan Details

Site Domain ftp.gnu.org
Base Domain gnu.org
Scan Status Ok
Last Scan2025-07-31T23:25:40+00:00
Next Scan 2025-08-30T23:25:40+00:00

Last Scan

Scanned2025-07-31T23:25:40+00:00
URL https://ftp.gnu.org/robots.txt
Domain IPs 2001:470:142:3::b, 209.51.188.20
Response IP 209.51.188.20
Found Yes
Hash b7fece6222aeb3feb5f1e9243616b1faf2790e2ff46597f8a9ecb668dc645b71
SimHash 301e9953dde2

Groups

*

Rule Path
Disallow /norobotsnorhumansshouldevervisithispage/
Allow /

Other Records

Field Value
crawl-delay 60

mj12bot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-ocob

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

oncrawl

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

academicbotrtu

Rule Path
Disallow /

timpibot

Rule Path
Disallow /
Back to top

Comments

  • robots.txt for https://ftp.gnu.org/
  • Majestic - SEO
  • DataForSeo - SEO
  • webmeup - SEO
  • Ahrefs - SEO
  • babbar - SEO
  • Screamingfrog - SEO
  • Seozoom - SEO
  • Brandwatch - SEO
  • Begin Moz - SEO
  • Not to be confused with Mozilla.
  • End Moz - SEO
  • Begin Semrush - SEO
  • End Semrush - SEO
  • cognitiveSEO - SEO
  • oncrawl - SEO
  • BEGIN Awario - Marketing
  • END Awario - Marketing
  • SERPSTAT - SEO
  • website-datenbank.de - Search engine?
  • Ignores crawl-delay and does not help us.
  • Aggressive Latvian Academic Integrity bot that does not help us.
  • Timpi NFT
Back to top