12
4GB to 8GB of RAM? That's not large by any standard?! And a bit less than 2 tokens per second for a 6B parameter model isn't fast at all.
4GB to 8GB of RAM? That's not large by any standard?! And a bit less than 2 tokens per second for a 6B parameter model isn't fast at all.
Who downvotes this
A community for the discussion of all single board computers. Raspberry Pi is ok, but there are so many other boards now that get looked over that deserve attention.
Post news, questions, your setups, guides, anything that has to do with SBCs. Server wide rules apply