Depends on the usecase. If you don’t need chat history for new-joiners, you can work with a single key per group, rotating it whenever someone joins or leaves. Since the server broadcasts a „so-and-so has left/joined“ it might as well include the new key. That key is then used by everyone in the group, so you can still broadcast all messages and don’t have to encrypt them individually.

Don’t let perfect be the enemy of good. Discord isn’t E2EE either. Having data under your control even if not encrypted is a big win.

If all your backups are near you, a flood or fire (or even break in) can still cost you all your data. At least one copy should be off-site.

For 7 people you could look into Virola Messenger. Not open source but uses Mumble under the hood and is super lightweight. No electron shit.

XMPP is also still a thing and IMO much easier to host (at least ejabberd is). Look into Movim, which looks quite nice as a discord replacement on top of XMPP.