| --- |
| license: apache-2.0 |
| tags: |
| - moe |
| - frankenmoe |
| - merge |
| - mergekit |
| - lazymergekit |
| - Gille/StrangeMerges_32-7B-slerp |
| - mlabonne/AlphaMonarch-7B |
| base_model: |
| - Gille/StrangeMerges_32-7B-slerp |
| - mlabonne/AlphaMonarch-7B |
| model-index: |
| - name: MixtureofMerges-MoE-2x7b-v7 |
| results: |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: AI2 Reasoning Challenge (25-Shot) |
| type: ai2_arc |
| config: ARC-Challenge |
| split: test |
| args: |
| num_few_shot: 25 |
| metrics: |
| - type: acc_norm |
| value: 73.21 |
| name: normalized accuracy |
| source: |
| url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jsfs11/MixtureofMerges-MoE-2x7b-v7 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: HellaSwag (10-Shot) |
| type: hellaswag |
| split: validation |
| args: |
| num_few_shot: 10 |
| metrics: |
| - type: acc_norm |
| value: 89.05 |
| name: normalized accuracy |
| source: |
| url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jsfs11/MixtureofMerges-MoE-2x7b-v7 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: MMLU (5-Shot) |
| type: cais/mmlu |
| config: all |
| split: test |
| args: |
| num_few_shot: 5 |
| metrics: |
| - type: acc |
| value: 64.63 |
| name: accuracy |
| source: |
| url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jsfs11/MixtureofMerges-MoE-2x7b-v7 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: TruthfulQA (0-shot) |
| type: truthful_qa |
| config: multiple_choice |
| split: validation |
| args: |
| num_few_shot: 0 |
| metrics: |
| - type: mc2 |
| value: 78.34 |
| source: |
| url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jsfs11/MixtureofMerges-MoE-2x7b-v7 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: Winogrande (5-shot) |
| type: winogrande |
| config: winogrande_xl |
| split: validation |
| args: |
| num_few_shot: 5 |
| metrics: |
| - type: acc |
| value: 84.93 |
| name: accuracy |
| source: |
| url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jsfs11/MixtureofMerges-MoE-2x7b-v7 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: GSM8k (5-shot) |
| type: gsm8k |
| config: main |
| split: test |
| args: |
| num_few_shot: 5 |
| metrics: |
| - type: acc |
| value: 69.07 |
| name: accuracy |
| source: |
| url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jsfs11/MixtureofMerges-MoE-2x7b-v7 |
| name: Open LLM Leaderboard |
| --- |
| |
| # MixtureofMerges-MoE-2x7b-v7 |
|
|
| MixtureofMerges-MoE-2x7b-v7 is a Mixture of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): |
| * [Gille/StrangeMerges_32-7B-slerp](https://huggingface.co/Gille/StrangeMerges_32-7B-slerp) |
| * [mlabonne/AlphaMonarch-7B](https://huggingface.co/mlabonne/AlphaMonarch-7B) |
|
|
| ## 🧩 Configuration |
|
|
| ```yaml |
| base_model: Gille/StrangeMerges_32-7B-slerp |
| gate_mode: hidden |
| dtype: bfloat16 |
| experts: |
| - source_model: Gille/StrangeMerges_32-7B-slerp |
| positive_prompts: |
| - "Answer this question from the ARC (Argument Reasoning Comprehension)." |
| - "Use common sense and logical reasoning skills." |
| - "What assumptions does this argument rely on?" |
| - "Are these assumptions valid? Explain." |
| - "Analyze the logical structure of this argument. Identify the premises, conclusion, and any assumptions made" |
| - "Identify any potential counterarguments to this position. How might someone challenge the reasoning presented?" |
| - "Could this be explained in a different way? Provide an alternative explanation." |
| - "Identify any weaknesses in this argument." |
| - "Does this argument contain any logical fallacies? If so, which ones?" |
| - "Generate a few possible continuations to this scenario." |
| - "Demonstrate understanding of everyday commonsense in your response." |
| - "Use contextual clues to determine the most likely outcome." |
| - "Continue this scenario, but make the writing style sound archaic and overly formal." |
| - "This narrative is predictable. Can you introduce an unexpected yet plausible twist?" |
| - "The character is angry. Continue this scenario showcasing a furious outburst." |
| negative_prompts: |
| - "misses key evidence" |
| - "overly general" |
| - "commits the fallacy of hasty generalization" |
| - "focuses on irrelevant details" |
| - "assumes information not provided" |
| - "relies on stereotypes" |
| - "repetitive phrases" |
| - "engages in circular reasoning" |
| - "overuse of the same words" |
| - "contradicts earlier statements - breaks the internal logic of the scenario" |
| - "out of character dialogue" |
| - "awkward phrasing - sounds unnatural" |
| - "doesn't match the given genre" |
| - source_model: mlabonne/AlphaMonarch-7B |
| positive_prompts: |
| - "Answer this question, demonstrating commonsense understanding and using any relevant general knowledge you may have." |
| - "Provide a concise summary of this passage, then explain why the highlighted section is essential to the main idea." |
| - "Read these two brief articles presenting different viewpoints on the same topic. List their key arguments and highlight where they disagree." |
| - "Paraphrase this statement, changing the emotional tone but keeping the core meaning intact. Example: Rephrase a worried statement in a humorous way" |
| - "Create a short analogy that helps illustrate the main concept of this article." |
| - "Explain the concept of physics to a high school student. Use analogies and examples to clarify the main ideas." |
| - "Calculate the answer to this math problem" |
| - "My mathematical capabilities are strong, allowing me to handle complex mathematical queries" |
| - "solve for" |
| - "Analyze the given data and identify any patterns or trends. What conclusions can be drawn from this information?" |
| - "A store sells apples at $0.50 each. If Emily buys 12 apples, how much does she need to pay?" |
| - "Isolate x in the following equation: 2x + 5 = 17" |
| - "Solve this equation and show your working." |
| - "Explain why you used this formula to solve the problem." |
| - "Attempt to divide this number by zero. Explain why this cannot be done." |
| negative_prompts: |
| - "sounds too basic" |
| - "understated" |
| - "dismisses important details" |
| - "avoids the question's nuance" |
| - "skips essential steps in the solution" |
| - "takes this statement too literally" |
| - "incorrect" |
| - "inaccurate" |
| - "assumed without proof" |
| - "uses jargon without explanation" |
| - "rushed calculation" |
| - "confuses mathematical concepts" |
| - "draws illogical conclusions" |
| - "circular reasoning" |
| ``` |
|
|
| ## 💻 Usage |
|
|
| ```python |
| !pip install -qU transformers bitsandbytes accelerate |
| |
| from transformers import AutoTokenizer |
| import transformers |
| import torch |
| |
| model = "jsfs11/MixtureofMerges-MoE-2x7b-v7" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model) |
| pipeline = transformers.pipeline( |
| "text-generation", |
| model=model, |
| model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True}, |
| ) |
| |
| messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}] |
| prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) |
| print(outputs[0]["generated_text"]) |
| ``` |
| # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
| Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_jsfs11__MixtureofMerges-MoE-2x7b-v7) |
|
|
| | Metric |Value| |
| |---------------------------------|----:| |
| |Avg. |76.54| |
| |AI2 Reasoning Challenge (25-Shot)|73.21| |
| |HellaSwag (10-Shot) |89.05| |
| |MMLU (5-Shot) |64.63| |
| |TruthfulQA (0-shot) |78.34| |
| |Winogrande (5-shot) |84.93| |
| |GSM8k (5-shot) |69.07| |
|
|
|
|
