Log In Sign Up

karp.lol
robots.txt

Robots Exclusion Standard data for karp.lol

Resource Scan

Scan Details

Site Domain karp.lol
Base Domain karp.lol
Scan Status Ok
Last Scan5/3/2025, 4:35:10 PM
Next Scan 6/2/2025, 4:35:10 PM

Last Scan

Scanned5/3/2025, 4:35:10 PM
URL https://karp.lol/robots.txt
Domain IPs 152.53.16.62, 2a0a:4cc0:1:1176::1
Response IP 152.53.16.62
Found Yes
Hash 1b776a600a2928a7ca620b80bcbc981dcd87c20fa55f39173ec958c2a7822911
SimHash 362f4b58c1e4

Groups

ai2bot
ai2bot-dolma
amazonbot
anthropic-ai
applebot
applebot-extended
brightbot 1.0
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
cohere-training-data-crawler
crawlspace
diffbot
duckassistbot
facebookbot
friendlycrawler
google-extended
googleother
googleother-image
googleother-video
gptbot
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
imgproxy
isscyberriskcrawler
kangaroo bot
meta-externalagent
meta-externalfetcher
novaact
oai-searchbot
omgili
omgilibot
operator
pangubot
perplexity-user
perplexitybot
petalbot
scrapy
semrushbot-ocob
semrushbot-swa
sidetrade indexer bot
timpibot
velenpublicwebcrawler
webzio-extended
youbot

Rule Path
Disallow /

awariorssbot
awariosmartbot
dataforseobot
magpie-crawler
meltwater
peer39_crawler
peer39_crawler/1.0
piplbot
scoop.it
seekr

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /auth/
Disallow /oauth/
Disallow /check_your_email
Disallow /wait_for_approval
Disallow /account_disabled
Disallow /signup
Disallow /fileserver/
Disallow /users/
Disallow /emoji/
Disallow /admin
Disallow /user
Disallow /settings/
Disallow /about/suspended
Disallow /.well-known/webfinger
Disallow /.well-known/nodeinfo
Disallow /nodeinfo/

Other Records

Field Value
crawl-delay 500
Back to top

Comments

  • GoToSocial robots.txt -- to edit, see internal/api/util/robots.go
  • More info @ https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • AI scrapers and the like.
  • https://github.com/ai-robots-txt/ai.robots.txt/
  • Marketing/SEO "intelligence" data scrapers
  • Well-known.dev crawler. Indexes stuff under /.well-known.
  • https://well-known.dev/about/
  • Rules for everything else.
  • API endpoints.
  • Auth/Sign in endpoints.
  • Fileserver/media.
  • Fedi S2S API endpoints.
  • Settings panels.
  • Domain blocklist.
  • Webfinger endpoint.
  • Disallow nodeinfo
Back to top