Log In Sign Up

gts.lapidak.is
robots.txt

Robots Exclusion Standard data for gts.lapidak.is

Resource Scan

Scan Details

Site Domain gts.lapidak.is
Base Domain lapidak.is
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-11-24T19:01:05+00:00
Next Scan 2026-02-22T19:01:05+00:00

Last Successful Scan

Scanned2025-04-06T04:39:10+00:00
URL https://gts.lapidak.is/robots.txt
Domain IPs 104.21.72.234, 172.67.155.173, 2606:4700:3032::6815:48ea, 2606:4700:3037::ac43:9bad
Response IP 104.21.72.234
Found Yes
Hash c5dd7a60e8be96c61817fa8a510d96ebc6d9029c294ce4760250c23aeb640afa
SimHash 362e4b58c1e4

Groups

ai2bot
ai2bot-dolma
amazonbot
anthropic-ai
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
cohere-training-data-crawler
diffbot
duckassistbot
facebookbot
friendlycrawler
google-extended
googleother
googleother-image
googleother-video
gptbot
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
isscyberriskcrawler
kangaroo bot
meta-externalagent
meta-externalfetcher
oai-searchbot
omgili
omgilibot
pangubot
perplexitybot
petalbot
scrapy
sidetrade indexer bot
timpibot
velenpublicwebcrawler
webzio-extended
youbot

Rule Path
Disallow /

awariorssbot
awariosmartbot
dataforseobot
magpie-crawler
meltwater
peer39_crawler
peer39_crawler/1.0
piplbot
scoop.it
seekr

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /auth/
Disallow /oauth/
Disallow /check_your_email
Disallow /wait_for_approval
Disallow /account_disabled
Disallow /signup
Disallow /fileserver/
Disallow /users/
Disallow /emoji/
Disallow /admin
Disallow /user
Disallow /settings/
Disallow /about/suspended
Disallow /.well-known/webfinger
Disallow /.well-known/nodeinfo
Disallow /nodeinfo/

Other Records

Field Value
crawl-delay 500
Back to top

Comments

  • GoToSocial robots.txt -- to edit, see internal/api/util/robots.go
  • More info @ https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • AI scrapers and the like.
  • https://github.com/ai-robots-txt/ai.robots.txt/
  • Marketing/SEO "intelligence" data scrapers
  • Well-known.dev crawler. Indexes stuff under /.well-known.
  • https://well-known.dev/about/
  • Rules for everything else.
  • API endpoints.
  • Auth/Sign in endpoints.
  • Fileserver/media.
  • Fedi S2S API endpoints.
  • Settings panels.
  • Domain blocklist.
  • Webfinger endpoint.
  • Disallow nodeinfo
Back to top