Returning to the original question
Between December 2023 and January 2024, I released nine FusionNet model variants on Hugging Face. They span dense, linear, passthrough, SOLAR, and two-expert mixture-of-experts compositions. The recorded model sizes range from 10.7B to 60.8B parameters.
The question behind this work was whether useful capabilities from existing language models could be combined into another model. Looking back, the more difficult question is how to determine what a composition changed. A stronger aggregate score is a useful observation, but it does not identify the source of the improvement.
This article is a retrospective on those releases, written for this research archive. The project dates refer to the model releases, not to an earlier publication of this article.
What was released
The public repositories contain model weights, configuration files, and model cards. Hugging Face records the original FusionNet repository as created on December 31, 2023, followed by the remaining variants during January 2024. The two v0.1 MoE variants were created on January 30.
| Artifact | Approximate parameter count | Mean reported in its model card |
|---|---|---|
| FusionNet | 10.7B | 74.38 |
| FusionNet 7B×2 MoE v0.1 | 12.9B | 76.16 |
| FusionNet 34B×2 MoE v0.1 | 60.8B | 77.38 |
The means above are from the legacy Open LLM Leaderboard evaluation recorded in the cards. They combine ARC, HellaSwag, MMLU, TruthfulQA, WinoGrande, and GSM8K, with the respective few-shot settings listed in the cards. These are historical results, not current leaderboard positions. The model names are labels: “7B×2” is not the same as an exact 14B-parameter count, because a composition can share components.
This table is also not an architecture-controlled comparison. The variants differ in size and composition, and the public cards do not supply a complete matched experiment against every source model. I therefore do not use the table to estimate a causal benefit from MoE or merging.
Look below the average
A useful example is the difference between the two MoE v0.1 cards. The larger model has a substantially higher recorded MMLU score, but the smaller model has higher HellaSwag and WinoGrande scores. The aggregate mean hides that variation.
| Recorded task score | 12.9B MoE v0.1 | 60.8B MoE v0.1 |
|---|---|---|
| HellaSwag, 10-shot | 88.90 | 86.46 |
| MMLU, 5-shot | 65.00 | 76.72 |
| WinoGrande, 5-shot | 87.53 | 83.35 |
| GSM8K, 5-shot | 70.28 | 73.01 |
The observation suggests that “better” needs a workload attached to it. It does not show that the smaller model is intrinsically better at a particular capability: the parent models, evaluation configuration, and sampling uncertainty still matter. It does show why a single arithmetic mean is too coarse to explain a composition experiment.
I would want to inspect examples on which a composition changes the answer relative to its parents, as well as the aggregate metrics. Are gains concentrated in one domain? Are errors correlated between the parents? Does the composed model preserve useful behavior outside the tasks used to select it? Those questions turn a release into a study of capability transfer.
The control experiment that matters
For a new composition study, I would freeze the exact revisions of the parent checkpoints, tokenizer, chat template, and evaluation harness. I would then evaluate the parents and the composed model under the same task settings. A prompt or tokenizer mismatch can otherwise look like a capability change.
The comparison should include an inexpensive selection baseline. If choosing one parent model already matches the composed model on the target workload, a larger composition needs a different justification. A routing or ensembling baseline can answer another question: does combining weights offer an advantage over combining predictions, after accounting for serving cost?
For an MoE model, I would report both total parameters and the computation activated per input, together with measured latency and memory. A model name or total parameter count does not by itself specify inference cost. This connects the earlier language-model work to the systems question I now spend more time on: how much useful behavior is obtained from the computation actually performed?
What I would record more carefully
The surviving public cards are evidence of released artifacts and recorded evaluations. They are not, on their own, complete experimental provenance. For a new study I would preserve the composition recipe, exact parent revisions, selection criteria, unsuccessful candidates, and evaluation outputs alongside the weights.
I would also separate a development set from the final assessment. Repeatedly selecting variants on the same public benchmark can adapt the project to that benchmark even without deliberately training on its test examples. A fresh evaluation distribution and a fixed selection budget would make a later improvement easier to interpret.
These are improvements I would make to the experimental record today. I am not claiming that every proposed control was part of the original releases. The durable lesson from FusionNet is the importance of turning a model artifact into an auditable comparison: what was combined, what changed, what did it cost, and which evidence would challenge the explanation?
Artifacts and sources
- TomGrc model collection, including all nine FusionNet releases.
- FusionNet model card.
- 12.9B MoE v0.1 model card.
- 60.8B MoE v0.1 model card.
- Hugging Face repository metadata, including repository creation dates.
The values in this note are transcribed from existing model cards. No models were re-evaluated for this retrospective.