Timon

At 4 bit or lower use IQ-type quant. The math is more advanced and quantization error is lower. You can double the context with -ctk q8_0 -ctv q8_0for virtually no loss of quality and speed.

New activity in unsloth/gemma-4-26B-A4B-it-GGUF 3 months ago

Gemma 4 seems to work best with high temperature for coding

👍 1

#21 opened 3 months ago by

Reverger

New activity in mistralai/Mistral-Small-4-119B-2603 4 months ago

Recommended sampler?

#4 opened 4 months ago by

mratsim

New activity in bartowski/Qwen_Qwen3.5-397B-A17B-GGUF 4 months ago

Older quants get in the way

#1 opened 4 months ago by

KeyboardMasher

New activity in ggml-org/gpt-oss-120b-GGUF 11 months ago

Error with built-in Web UI

#3 opened 11 months ago by

KeyboardMasher

New activity in legraphista/Llama-3.2-3B-Instruct-IMat-GGUF about 1 year ago

Thanks for IQ4_NL

❤️ 1

#1 opened about 1 year ago by

KeyboardMasher

New activity in bartowski/Qwen_Qwen3-30B-A3B-GGUF about 1 year ago

128k Context GGUF, please?

#2 opened about 1 year ago by

MikeNate

New activity in bartowski/inclusionAI_Ling-lite-0415-GGUF about 1 year ago

Update README.md

#1 opened about 1 year ago by

KeyboardMasher

New activity in bartowski/google_gemma-3-27b-it-qat-GGUF about 1 year ago

Other Imatrix quants (IQ3_XS) ?

👍 3

#1 opened about 1 year ago by deleted

reacted to bartowski's post with 👍 about 1 year ago

Post

39913

Access requests enabled for latest GLM models

While a fix is being implemented (https://github.com/ggml-org/llama.cpp/pull/12957) I want to leave the models up for visibility and continued discussion, but want to prevent accidental downloads of known broken models (even though there are settings that could fix it at runtime for now)

With this goal, I've enabled access requests. I don't really want your data, so I'm sorry that I don't think there's a way around that? But that's what I'm gonna do for now, and I'll remove the gate when a fix is up and verified and I have a chance to re-convert and quantize!

Hope you don't mind in the mean time :D

1 reply

upvoted an article over 1 year ago

Article

Comparing sub 50GB Llama 4 Scout quants (KLD/Top P)

bartowski

•

Apr 9, 2025

• 45

New activity in bartowski/QVQ-72B-Preview-GGUF over 1 year ago

llama.cpp inference too slow?

#6 opened over 1 year ago by

ygsun

New activity in unsloth/DeepSeek-R1-GGUF over 1 year ago

Over 2 tok/sec agg backed by NVMe SSD on 96GB RAM + 24GB VRAM AM5 rig with llama.cpp

👍🔥 4

#13 opened over 1 year ago by

ubergarm

reacted to fdaudens's post with 😎 over 1 year ago

Post

10152

Yes, DeepSeek R1's release is impressive. But the real story is what happened in just 7 days after:

- Original release: 8 models, 540K downloads. Just the beginning...

- The community turned those open-weight models into +550 NEW models on Hugging Face. Total downloads? 2.5M—nearly 5X the originals.

The reason? DeepSeek models are open-weight, letting anyone build on top of them. Interesting to note that the community focused on quantized versions for better efficiency & accessibility. They want models that use less memory, run faster, and are more energy-efficient.

When you empower builders, innovation explodes. For everyone. 🚀

The most popular community model? @bartowski 's DeepSeek-R1-Distill-Qwen-32B-GGUF version — 1M downloads alone.