Log In Sign Up

toot.lt
robots.txt

Robots Exclusion Standard data for toot.lt

Resource Scan

Scan Details

Site Domain toot.lt
Base Domain toot.lt
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-11-28T22:36:31+00:00
Next Scan 2025-12-28T22:36:31+00:00

Last Successful Scan

Scanned2025-09-14T12:48:34+00:00
URL https://toot.lt/robots.txt
Redirect https://gts.poci.us/robots.txt
Redirect Domain gts.poci.us
Redirect Base poci.us
Domain IPs 104.21.50.121, 172.67.163.5, 2606:4700:3037::6815:3279, 2606:4700:3037::ac43:a305
Redirect IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.64.1
Found Yes
Hash 1b776a600a2928a7ca620b80bcbc981dcd87c20fa55f39173ec958c2a7822911
SimHash 362f4b58c1e4

Groups

ai2bot
ai2bot-dolma
amazonbot
anthropic-ai
applebot
applebot-extended
brightbot 1.0
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
cohere-training-data-crawler
crawlspace
diffbot
duckassistbot
facebookbot
friendlycrawler
google-extended
googleother
googleother-image
googleother-video
gptbot
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
imgproxy
isscyberriskcrawler
kangaroo bot
meta-externalagent
meta-externalfetcher
novaact
oai-searchbot
omgili
omgilibot
operator
pangubot
perplexity-user
perplexitybot
petalbot
scrapy
semrushbot-ocob
semrushbot-swa
sidetrade indexer bot
timpibot
velenpublicwebcrawler
webzio-extended
youbot

Rule Path
Disallow /

awariorssbot
awariosmartbot
dataforseobot
magpie-crawler
meltwater
peer39_crawler
peer39_crawler/1.0
piplbot
scoop.it
seekr

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /auth/
Disallow /oauth/
Disallow /check_your_email
Disallow /wait_for_approval
Disallow /account_disabled
Disallow /signup
Disallow /fileserver/
Disallow /users/
Disallow /emoji/
Disallow /admin
Disallow /user
Disallow /settings/
Disallow /about/suspended
Disallow /.well-known/webfinger
Disallow /.well-known/nodeinfo
Disallow /nodeinfo/

Other Records

Field Value
crawl-delay 500
Back to top

Comments

  • GoToSocial robots.txt -- to edit, see internal/api/util/robots.go
  • More info @ https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • AI scrapers and the like.
  • https://github.com/ai-robots-txt/ai.robots.txt/
  • Marketing/SEO "intelligence" data scrapers
  • Well-known.dev crawler. Indexes stuff under /.well-known.
  • https://well-known.dev/about/
  • Rules for everything else.
  • API endpoints.
  • Auth/Sign in endpoints.
  • Fileserver/media.
  • Fedi S2S API endpoints.
  • Settings panels.
  • Domain blocklist.
  • Webfinger endpoint.
  • Disallow nodeinfo
Back to top