75
Llama 3.1 AI Models Have Officially Released
(llama.meta.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 23 Jul 2024
75 points (89.5% liked)
Technology
59419 readers
2874 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
[Oh, my friend, you have to switch to this: https://huggingface.co/BeaverAI/mistral-doryV2-12b
It's so much smarter than llama 13B. And it goes all the way out to 128K!
Oof - not on my 12gb 3060 it doesn't :/ Even at 48k context and the Q4_K quantization, it's ollama its doing a lot of offloading to the cpu. What kind of hardware are you running it on?
A 3090.
But it should be fine on a 3060, with zero offloading.
Dump ollama for long context. Grab a 5-6bpw exl2 quantization and load it with Q4 or Q6 cache depending on how much context you want. I personally use EXUI, but text-gen-webui and tabbyapi (with some other frontend) will also load them.