nicolasembleton/context-1-GGUF

Context-1 GGUF Quantizations

GGUF quantized versions of chromadb/context-1, converted for inference with llama.cpp, LM Studio, and other GGUF-compatible engines.

About Context-1

Context-1 is a 20.9B parameter Mixture-of-Experts (MoE) causal language model developed by Chroma. It uses the GptOssForCausalLM architecture with 32 experts and 4 active per token, providing strong performance with efficient inference.

Detail	Value
Architecture	GptOssForCausalLM (MoE)
Total Parameters	~20.9B
Active Parameters	~3B per token (4 of 32 experts)
Hidden Size	2880
License	Apache-2.0

Quantization

Quantized from F16 weights using llama.cpp with importance matrix (imatrix) calibration, running on NVIDIA H100 GPUs via Modal. All standard K-quant and I-quant variants are provided.

Usage

llama.cpp

# Download your preferred quant
huggingface-cli download nicolasembleton/context-1-GGUF context-1-Q4_K_M.gguf --local-dir .

# Run
./llama-cli -m context-1-Q4_K_M.gguf -p "Your prompt here" -ngl 99

LM Studio

Search for nicolasembleton/context-1-GGUF in LM Studio's model browser and download the desired quantization.

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="nicolasembleton/context-1-GGUF",
    filename="context-1-Q4_K_M.gguf",
    n_gpu_layers=-1,
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response)

Chat Template

This model uses a custom chat template based on the OpenAI/Oss architecture with support for multi-channel output (analysis, commentary, final), tool calling, and built-in browser/python tools. The template is embedded in the GGUF files.

Format overview:

<|start|>system<|message|>...<|end|>
<|start|>developer<|message|>...<|end|>
<|start|>user<|message|>...<|end|>
<|start|>assistant<|channel|>final<|message|>...<|end|>

For the full template, see chat_template.jinja in the original repository.

License

Apache-2.0 — same as the original model.

Acknowledgements

• Chroma for the original model
• llama.cpp for the quantization tooling
• Modal for the GPU compute infrastructure