Log In Sign Up

fossies.org
robots.txt

Robots Exclusion Standard data for fossies.org

Resource Scan

Scan Details

Site Domain fossies.org
Base Domain fossies.org
Scan Status Ok
Last Scan2024-08-31T01:09:06+00:00
Next Scan 2024-09-30T01:09:06+00:00

Last Scan

Scanned2024-08-31T01:09:06+00:00
URL https://fossies.org/robots.txt
Domain IPs 148.251.50.230
Response IP 148.251.50.230
Found Yes
Hash 4458f98b90e3916fe929f68f8592a044f67c2cbf704576b00cd31168f9c57bca
SimHash 429c10d2ccb0

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yacybot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

hubspot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

linguee

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

mtrobot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

paracrawl

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

*

Rule Path
Disallow /linux/misc/legacy/
Allow /linux/misc/legacy/index.html
Disallow /linux/www/legacy/
Allow /linux/www/legacy/index.html
Disallow /linux/privat/legacy/
Allow /linux/privat/legacy/index.html
Back to top

Other Records

Field Value
sitemap https://fossies.org/sitemaps/sitemap.index.xml.gz
Back to top

Comments

  • robots.txt file for https://fossies.org/ (and http://fresh-center.com/)
  • Please contact admin@fossies.org with concerns.
Back to top