Log In Sign Up

fvnki.town
robots.txt

Robots Exclusion Standard data for fvnki.town

Resource Scan

Scan Details

Site Domain fvnki.town
Base Domain fvnki.town
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2025-12-12T13:55:31+00:00
Next Scan 2026-03-12T13:55:31+00:00

Last Successful Scan

Scanned2023-11-17T01:32:40+00:00
URL https://fvnki.town/robots.txt
Domain IPs 46.139.3.54
Response IP 46.139.3.54
Found Yes
Hash 76892531dbef8c5f3988713d3dc8db9cc84c661241f7cadaa7d698bd50e3ce29
SimHash 783bdb1ecd4c

Groups

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /auth/
Disallow /oauth/
Disallow /check_your_email
Disallow /wait_for_approval
Disallow /account_disabled
Disallow /.well-known/
Disallow /fileserver/
Disallow /users/
Disallow /emoji/
Disallow /admin
Disallow /user
Disallow /settings/
Disallow /about/suspended

Other Records

Field Value
crawl-delay 500
Back to top

Comments

  • GoToSocial robots.txt -- to edit, see internal/web/robots.go
  • More info @ https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • Before we commence, a giant fuck you to ChatGPT in particular.
  • https://platform.openai.com/docs/gptbot
  • As of September 2023, GPTBot and ChatGPT-User are equivalent. But there's no telling
  • when OpenAI might decide to change that, so block this one too.
  • And a giant fuck you to Google Bard and their other generative AI ventures too.
  • https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
  • Block CommonCrawl. Used in training LLMs and specifically GPT-3.
  • https://commoncrawl.org/faq
  • Block Omgilike/Webz.io, a "Big Web Data" engine.
  • https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/
  • Block Faceboobot, because Meta.
  • https://developers.facebook.com/docs/sharing/bot
  • Well-known.dev crawler. Indexes stuff under /.well-known.
  • https://well-known.dev/about/
  • Rules for everything else.
  • API endpoints.
  • Auth/login endpoints.
  • Well-known endpoints.
  • Fileserver/media.
  • Fedi S2S API endpoints.
  • Settings panels.
  • Domain blocklist.
Back to top