Log In Sign Up

nostter.app
robots.txt

Robots Exclusion Standard data for nostter.app

Resource Scan

Scan Details

Site Domain nostter.app
Base Domain nostter.app
Scan Status Ok
Last Scan2025-07-28T00:19:10+00:00
Next Scan 2025-08-27T00:19:10+00:00

Last Scan

Scanned2025-07-28T00:19:10+00:00
URL https://nostter.app/robots.txt
Domain IPs 104.21.26.219, 172.67.139.116, 2606:4700:3030::6815:1adb, 2606:4700:3031::ac43:8b74
Response IP 172.67.139.116
Found Yes
Hash 02fa193013e0bb4a2518113a4443dfa779a2be6461e09f03b7839052a5ead23f
SimHash 73189402a8e4

Groups

facebookexternalhit
meta-externalagent
imagesiftbot
gptbot
googleother
mj12bot
claudebot
amazonbot

Rule Path
Disallow /npub
Disallow /nprofile
Disallow /note
Disallow /nevent
Disallow /naddr

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60
Back to top

Comments

  • https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
  • https://imagesift.com/about
  • https://platform.openai.com/docs/bots
  • https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
  • https://www.mj12bot.com/
  • https://support.anthropic.com/en/articles/10023637-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  • https://developer.amazon.com/support/amazonbot
Back to top