Update README.md

14981d6 verified 4 days ago

8.24 kB

license: mit
tags:
  - text-generation
  - causal-lm
  - grpo
  - reasoning
  - reinforcement-learning
  - mini-llm
  - text-generation-inference
  - lm-evaluation-harness
datasets:
  - openai/gsm8k
  - openai/openai_humaneval
  - cais/mmlu
  - allenai/ai2_arc
  - bespokelabs/Bespoke-Stratos-17k
language:
  - en
pipeline_tag: text-generation
metrics:
  - accuracy
  - exact_match
model-index:
  - name: Atomight-V2.1-0.5B-Inference
    results:
      - task:
          type: text-generation
          name: ARC-Challenge
        dataset:
          name: AI2 ARC-Challenge
          type: allenai/ai2_arc
          config: ARC-Challenge
          split: test
        metrics:
          - name: Accuracy
            type: acc
            value: 0.30802
          - name: Accuracy (Norm)
            type: acc_norm
            value: 0.33788
      - task:
          type: text-generation
          name: GSM8K
        dataset:
          name: GSM8K
          type: openai/gsm8k
          config: main
          split: test
        metrics:
          - name: Exact Match (Strict)
            type: exact_match
            value: 0.19788
          - name: Exact Match (Flexible)
            type: exact_match
            value: 0.32449
      - task:
          type: text-generation
          name: ARC-Easy
        dataset:
          name: AI2 ARC-Easy
          type: allenai/ai2_arc
          config: ARC-Easy
          split: test
        metrics:
          - name: Accuracy
            type: acc
            value: 0.65446
          - name: Accuracy (Norm)
            type: acc_norm
            value: 0.59343
      - task:
          type: text-generation
          name: HellaSwag
        dataset:
          name: HellaSwag
          type: Rowan/hellaswag
          split: validation
        metrics:
          - name: Accuracy
            type: acc
            value: 0.40709
          - name: Accuracy (Norm)
            type: acc_norm
            value: 0.5235

Atomight-V2.1-0.5B-Inference

Atomight Logo

Atomight-V2.1-0.5B-Inference is an ultra-compact, reasoning-oriented causal language model developed under the Atomight Ecosystem. Built on a Qwen-derived 494M parameter foundation, the model has been refined using GRPO (Group Relative Policy Optimization) reinforcement tuning.

Despite its tiny physical footprint, Atomight-V2.1-0.5B targets highly efficient edge-device reasoning, structured text outputs, lightweight coding assistance, and rapid deployment workflows under severe compute constraints.

🚀 Key Highlights

Parameter Footprint: ~494M parameters (Loads into ~1GB VRAM at FP16).
Training Paradigm: GRPO reinforcement learning focusing on high-signal reasoning vectors instead of brute-force dataset scale.
Edge-Optimized: Designed specifically for low-overhead mobile, local, and browser-based inference loops (Google Colab / Kaggle native workflow).

Weighted/Imatrix Quants, and Static Quants by mradermacher are available at: https://huggingface.co/mradermacher/Atomight-V2.1-0.5B-Inference-i1-GGUF https://huggingface.co/mradermacher/Atomight-V2.1-0.5B-Inference-GGUF

NOTICE: Think tags (chain-of-thought reasoning) may not be working now due to some errors. But rest assured, the initial model is actively working & SAFE. We'll try our best to fix this as soon as possible.

📊 Evaluation & Benchmark Results

Official evaluations were conducted using the EleutherAI LM Evaluation Harness at FP16 precision.

Core Evaluation Metrics

Benchmark Task	Metric Typology	Atomight-V2.1-0.5B Score	Focus Domain
ARC-Easy	Accuracy (Normalized)	59.34%	Scientific Fact Retrieval
HellaSwag	Accuracy (Normalized)	52.35%	Commonsense Reasoning & Next-Sentence Prediction
ARC-Challenge	Accuracy (Normalized)	33.79%	Hard Analytical Exclusion Logic
GSM8K (Flexible Extract)	Exact Match (Regex Clean)	32.45%	Mathematical Thought & Resolution
GSM8K (Strict)	Exact Match (Rigid Parse)	19.79%	Formatted Mathematical Output

Atomight V2.1 Benchmark

🔍 Comparative Engineering Insights

Punching Above Weight Classes: Atomight-V2.1-0.5B outpaces Meta's larger Llama-3.2-1B-Instruct on localized logic-retrieval metrics, clearing 59.3% on ARC-Easy and 33.8% on ARC-Challenge compared to Llama's 56.7% and 31.8% respectively.
The Reasoning Gap: On mathematical reasoning (GSM8K), when evaluated with Flexible Extraction parsing (32.45%), Atomight demonstrates higher raw mathematical accuracy than both Qwen2.5-0.5B-Instruct (26.8%) and Llama-3.2-1B-Instruct (24.4%).
The Formatting Note: The delta between Atomight's Strict Math score (19.8%) and Flexible Math score (32.5%) stems from the internal reasoning tokens generated during the inference step. While the mathematical conclusion is correct nearly 1/3 of the time, the model frequently bypasses rigid formatting constraints in favor of dense thinking traces.

💻 Quickstart: Inference Execution

Atomight utilizes system and sequence prompts to partition thinking spaces. For optimal reasoning convergence, use explicit <thinking> and <answer> encapsulation layers.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NovatasticRoScript/Atomight-V2.1-0.5B-Inference"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    device_map="auto"
)

# Structuring system guidelines for GRPO activation
messages = [
    {
        "role": "system", 
        "content": "You are a reasoning model. Think inside <thinking> and answer inside <answer>."
    },
    {
        "role": "user", 
        "content": "A farmer has 12 apples. He gives 4 to his neighbor and loses 2 on the way home. How many apples does he have left?"
    }
]

inputs = tokenizer.apply_chat_template(
    messages, 
    tokenize=True, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        inputs, 
        max_new_tokens=250, 
        temperature=0.01,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

## 📄 Citations

@misc{shao2024deepseekmathpushinglimitsmathematical,
      title={DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}, 
      author={Zhihong Shao and Peiyi Wang and Qihao Zhu and Junxiao Song and Daya Guo and et al.},
      year={2024},
      eprint={2402.03300},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2402.03300}
}

@misc{qwen_qwen25_2025,
      title={Qwen2.5 Technical Report}, 
      author={Qwen and Yang, An and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and et al.},
      year={2024},
      eprint={2412.15115},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.15115}
}

@misc{eval-harness,
      author={Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and et al.},
      title={A framework for few-shot language model evaluation},
      month={12},
      year={2023},
      publisher={Zenodo},
      version={v0.4.0},
      doi={10.5281/zenodo.10257521},
      url={https://github.com/EleutherAI/lm-evaluation-harness}
}

@article{allenai:arc,
      author    = {Peter Clark  and Isaac Cowhey and Oren Etzioni and Tushar Khot and
                    Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},
      title     = {Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},
      journal   = {arXiv:1803.05457v1},
      year      = {2018},
}

@article{cobbe2021gsm8k,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
  journal={arXiv preprint arXiv:2110.14168},
  year={2021}
}