• 1 Post
  • 108 Comments
Joined 2 years ago
cake
Cake day: July 4th, 2023

help-circle


  • I know you said consumer GPU, but I run a used Tesla P40. It has 24 GB of vram. The price has gone up since I got it a couple years ago, there might be better options in the same price category. Still, it’s going to be cheaper than a modern full fat consumer gpu, with a reasonable performance hit.

    My use case is text generation, chat kind of things. In most cases, the inference is more than fast enough, but it can get slow when swapping out large context lengths.

    Mostly I run quantized 8-20B models with the sweet spot being around 12. For specialized use cases outside of general language, you can run more compact models. The general output is quite good, and I would have never had thought it was possible 10 years ago.

    ETA: I paid about $200 USD for the P40 a couple years ago, plus the price for a fan and 3d printed shroud.


  • I would do FDE yeah. My current laptop setup is with systemd-boot and a special initramfs that allows me to unlock it with a yubikey, with fallback to password. Fair warning, this exact configuration is not particularly easy to setup.

    There are also modules which enable early network connectivity along with a SSH server, meaning you login and unlock it remotely. I have not tried this.

    Debian does not frequently require rebooting under normal circumstances. Kernel updates are not that frequent, and you can usually put it off for a bit if you don’t want to deal with it.









  • Yeah, you can turn off registration without a token. Then, if you want someone to register you can issue them a registration token, or manually create their account.

    Federation can be turned on, on a case by case basis.

    You can set rooms to invite only and not discoverable. Alternately, you can use an invite-only space that allows users to join rooms from there.

    The first two parts are done in the server config, see the synapse docs. The last is done once the server is setup and running as an admin.