Exploring Hermes MoA: Is the Agg Or Ref Model More Important?

Does it matter which model you make the aggregator versus the reference in a Mixture of Agents setup? I built a deterministic eval suite to find out. 🧠 Run agents? Give them a knowledge base. Agent Wikis has free, curated LLM wikis on a ton of AI topics — plus a Pro tier ($9.99/mo) for super-sized wikis with way more depth: https://agentwikis.com/ Sign up for my FREE weekly newsletter, where I spill my unfiltered thoughts on the latest AI news, cool research, and projects I'm building: https://www.onchainaigarage.com/ A follow-up to my first Mixture of Agents video, this time with a real eval suite I built myself — deterministic Python scoring (no LLM grading its own homework) across speed, tool calls (BFCL), reasoning, instruction-following (IFEval), and a build-a-landing-page taste test. I run the same benchmark on MiMo v2.5 Pro and DeepSeek V4 Flash swapped between the aggregator and reference roles, plus each model solo, to isolate what the aggregator-vs-reference choice actually changes. The real question: does flipping which model drives versus advises make a measurable difference — and is an MoA even worth the extra time? Resources: 🔗 Hermes Agent: https://github.com/NousResearch/herme... Timestamps: 0:00 - MoA + the plan: aggregator vs reference 1:23 - The eval suite: 5 axes, deterministic scoring 3:04 - Configuring the MoA in the dashboard 4:50 - Run 1: Memo aggregator + DeepSeek reference 7:47 - Flipping it: DeepSeek aggregator + Memo reference 10:20 - Adding the solo-model baselines 11:23 - The results + what the aggregator choice does 12:41 - Open questions + next experiments #MixtureOfAgents #HermesAgent #AIAgents #NousResearch #LLM #DeepSeek #AIEval #MoA #AITools