22
submitted 9 months ago by gunpachi to c/localllama@sh.itjust.works

I have an rx 6600 and 16gb of ram and an i5 10400f

I am using oobabooga web-ui and I happened to have a gguf file of LLama2-13B-Tiefighter.Q4_K_S .

But it always says that the connection errored out when I load the model.

Anyway, please suggest any good model that I can get started with.

top 3 comments
sorted by: hot top controversial new old
[-] neurogenesis@lemmy.dbzer0.com 3 points 9 months ago

I'd suggest checking out WolframRavenwolf on raddit, he does regular LLM tests.

I'm looking at Beyonder 4x7B, Mistral Instruct 2x7B, Laser Dolphin 2x7B, and previously used Una Cybertron.

[-] gunpachi 3 points 9 months ago

Hey thanks ! I'll check these out.

[-] Dragonish@lemmy.dbzer0.com 3 points 9 months ago

I am using Mistral 7b Instruct for text summary and some light "assistant" type chatting for the last several months. I have been pleased at how accurate it is for my needs, especially given it's size.

I recall alot of trial and error to find models that were compatible with the version of llama-cpp-python that oobabooga uses (at any given time). GGUF should have made the model format (and therefore model selection) more simple, but i imagine there are still nuances that make it more difficult than it should be to find a working model for a noob.

Best of luck, let us know how it goes

this post was submitted on 25 Jan 2024
22 points (100.0% liked)

LocalLLaMA

2244 readers
1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago
MODERATORS