Log In Sign Up

gts-dev.pixie.town
robots.txt

Robots Exclusion Standard data for gts-dev.pixie.town

Resource Scan

Scan Details

Site Domain gts-dev.pixie.town
Base Domain pixie.town
Scan Status Ok
Last Scan2024-10-02T09:34:39+00:00
Next Scan 2024-10-03T09:34:39+00:00

Last Scan

Scanned2024-10-02T09:34:39+00:00
URL https://gts-dev.pixie.town/robots.txt
Domain IPs 159.69.36.55, 2a01:4f8:1c1c:7bbf::1
Response IP 159.69.36.55
Found Yes
Hash afa074923826f0ab231837ef22c7f897ec0cef7c64407932ff848a6cc68d81c4
SimHash 6826cb12cd5f

Groups

gptbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /auth/
Disallow /oauth/
Disallow /check_your_email
Disallow /wait_for_approval
Disallow /account_disabled
Disallow /.well-known/
Disallow /fileserver/
Disallow /users/
Disallow /emoji/
Disallow /admin
Disallow /user
Disallow /settings/
Disallow /about/suspended

Other Records

Field Value
crawl-delay 500
Back to top

Comments

  • GoToSocial robots.txt -- to edit, see internal/web/robots.go
  • More info @ https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • Before we commence, a giant fuck you to ChatGPT in particular.
  • https://platform.openai.com/docs/gptbot
  • Rules for everything else.
  • API endpoints.
  • Auth/login endpoints.
  • Well-known endpoints.
  • Fileserver/media.
  • Fedi S2S API endpoints.
  • Settings panels.
  • Domain blocklist.
Back to top