LocalLLaMA

2732 readers

16 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago

MODERATORS

pax@sh.itjust.works

SkySyrup@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

Loaded benchmark for 1-3-4-7b models? (lemm.ee)

submitted 4 days ago by thickertoofan@lemm.ee to c/localllama@sh.itjust.works

4 comments fedilink hide all child comments

I don't care a lot about mathematical tasks, but code intellingence is a minor preference but the most anticipated one is overall comprehension, intelligence. (For RAG and large context handling) But anyways any benchmark with a wide variety of models is something I am searching for, + updated.

you are viewing a single comment's thread
view the rest of the comments

[–] Smokeydope@lemmy.world 2 points 3 days ago* (last edited 3 days ago) (1 children)

Cool, page assist looks neat I'll have to check it out sometimes. My llm engine is kobold.cpp, and I just user the openwebui in internet browser to connect.

Sorry I don't really have good suggestions for you beyond to just try some of the more popular 1-4bs in a very high quant if not full f8 and see which works best for your use case.

Llama 4b, mistral 4b, phi-3-mini, tinyllm 1.5b, qwen 2-1.5b, ect ect. I assume you want a model with large context size and good comprehension skills to summarize youtube transcripts and webpage articles? At least I think thats what the add-on you mentioned suggested was its purpose.

So look for models with those things over ones that try to specialize in a little bit of domain knowledge.

[–] thickertoofan@lemm.ee 2 points 2 days ago

I checked mostly all of em out from the list, but 1b models are generally unusable for RAG.