Log In Sign Up

fedi.tilde.green
robots.txt

Robots Exclusion Standard data for fedi.tilde.green

Resource Scan

Scan Details

Site Domain fedi.tilde.green
Base Domain tilde.green
Scan Status Ok
Last Scan2025-06-07T21:33:54+00:00
Next Scan 2025-06-08T21:33:54+00:00

Last Scan

Scanned2025-06-07T21:33:54+00:00
URL https://fedi.tilde.green/robots.txt
Domain IPs 2001:41d0:303:5ddb::, 54.38.176.219
Response IP 54.38.176.219
Found Yes
Hash 9bc7f6a7c8eeccd64532830e40846741a25506b67bfb0b0ca8b86f9c0b248117
SimHash bad1db00c563

Groups

*

Rule Path
Disallow /noindex/

wibybot

Rule Path
Allow /

search.marginalia.nu

Rule Path
Allow /

adsbot

Rule Path
Disallow /
Allow /ads.txt
Allow /app-ads.txt

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

npbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

slysearch

Rule Path
Disallow /

checkmarknetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)

Rule Path
Disallow /

brandverity/1.0

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

cotoyogi

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

genai

Rule Path
Disallow /

semrushbot-ocob

Rule Path
Disallow /

semrushbot-ft

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /
Back to top

Comments

  • Based on:
  • - https://seirdy.one/meta/scrapers-i-block/
  • - https://seirdy.one/robots.txt
  • Sketchy
  • AI
Back to top