Selective Escalation: Phase 1

Harness escalation using heterogeneous hardware and mixtures of model families

Not all tokens are generated equally. Not all problems should be solved on foundation models. 

In this research project, we developed an agentic harness using Furiosa RNGD and AMD MI300X accelerators to explore whether we could retain the baseline quality of a foundation model running on the MI300Xs while offloading simpler problems to smaller models running on the Furiosa NPU accelerators. 

We call this selective escalation: reviewing a problem/prompt and choosing the right model/silicon combination to reduce both operating cost (power) and token generation costs.  

This blog reviews Phase 1 of our research at a high level. Phase 2 will focus on experimentation that improves the quality to understand the trade-offs needed. 

For the qualitative assessment, we used LiveCodeBench v4 and v5 for recursive training of the harness via Open Knowledge Format v0.2 (OKF), Agent Skills, and tool calling. Final benchmark sweeps included LiveCodeBench v6.

Here’s the setup: 

• 4x Furiosa RNGD NPUs running GPT-OSS-120B as a coder 

• 4x Furiosa RNGD NPUs running GPT-OSS-120B as a reviewer 

• 8x AMD MI300X GPUs running DeepSeek v4.1-Flash for planning, reasoning, and “hard” tasks the reviewer asserts require DeepSeek

As an aside, the I/ONX team does not have any harness engineering staff or extensive experience with these systems. We fully expect teams with proficiency to far exceed the results of our findings. 

At a high-level, here’s the workflow for our harness: 

We have a detailed diagram here

Snapshot

Phase 1 Goals

For our first phase of this research we wanted to understand if a harness could be leveraged to selectively allocate certain tasks to models running on the most cost-effective accelerators without compromising quality. 

System Specifications

I/ONX built a transparent bridging PCIe-fabric that enables up to 64 accelerators to fully enumerate on a host machine. Our host machine has dual EPYC Genoa-class CPUs (256 cores), 3TB system memory, and 50TB NVMe storage. 

All 16 accelerators are fully enumerated (no virtualization) on the same system for this research: 

$ lspci | grep -iE "mi300|furiosa"

07:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

0a:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

12:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

18:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

1e:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

21:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

29:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

2f:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X]

8b:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

8c:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

8d:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

8e:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

94:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

95:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

96:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

97:00.0 Processing accelerators: FuriosaAI, Inc. RNGD (rev 01)

Harness 

The harness was recursively trained via the v4 and v5 questions. We held back on the v6 questions to prevent any pollution of the OKF content. The recursive loop included instructions to never incorporate answers to questions and we reviewed the OKF content post-training to confirm no answers were present. 

Baseline

First, we set our baseline with DeepSeek running at 1M token context window: 

• Correct answers: 151

• Test time: 6.7 hours

• Tokens: 7.47 million

These results align with other publicly reported values for DeepSeek at 1M context window length. The 3rd party results we identified only disclose the number of correct answers. 

Phase 1 Results

• Correct answers: 153 - Same, within variance 

• Test time: 2.6 hours - 60% reduction 

• DeepSeek Tokens: 1M - 86% reduction 

The harness successfully maintained the quality for LiveCodeBench on v6 question sets while dramatically reducing the time and tokens generated by DeepSeek v4.1-Flash. We observed some volatility in the number of correct responses during, which is attributed to the methodology of LiveCodeBench hidden answer sets. 

Discussion

The Furiosa RNGD devices were selected because of their lower power profile and ability to run a model class in the 120B range. At 180W and their price point, they represent an extremely attractive accelerator profile for medium-sized large language models. Additional models were tested, but GPT-OSS-120B scored the highest for those questions the harness selected. 

The AMD MI300X was chosen because of its large amount of HBM memory - 192GB per GPU - necessary for DeepSeek v4.1-Flash if you want the entirety of the weights in memory on the accelerator. 

The OKF and Agent Skills files improved the scoring for approximately 5-10 questions, but this was primarily attributed to more effective routing decisions. 

The results highlight that a harness can operate on heterogeneous model families on heterogeneous hardware, maintain quality, and drastically reduce the cost of tokens generated by foundation models. 

We do not have a price point for GPT-OSS-120B running on Furiosa devices, so we extrapolated the pricing from publicly available sources. We did the same for the DeepSeek v4.1-Flash pricing. 

The result is that the harness saved approximately 49% using TogetherAI for the pricing: 

Phase 2 Research

For our next phase, we are interested in improving the LiveCodeBench scores across the hard questions both on the Furiosa and the AMD accelerators. We plan to experiment with different models on the Furiosa devices in particular. GPT-OSS-120B was a clear winner as the coding agent, but the reviewer role may benefit from a different model. 

The main challenge with LiveCodeBench is algorithm selection, so a reviewer with a tuned OKF may provide better insights and instructions to the coder agent, for example. Additional research in harness-specific improvements, such as the OKF, are areas of interest. 

Ultimately, even 3-5 more hard answers solved correctly would be a significant improvement considering the amount and cost of tokens we have achieved, much less the time-to-completion improvements with our Phase 1 results over DeepSeek alone. 

Future Research

LiveCodeBench is a unique benchmark and our recursive loop approach worked well to stabilize the quality metrics. A future experiment would be to see how the harness generalizes to other qualitative benchmarks, such as Terminal-Bench, SWE-Bench, and SciCode. 

Another area of research is scheduling. For example, the larger the KV cache, the more important the scheduler. Moving caches from HBM to system memory to NVMe, to other storage appliances can free resources meaningfully when properly scheduled. 

Recent research shows promise for cache-to-cache communications across model families. Right now, the harness communicates with full prompt requests from model to model, which requires expensive prefill. Leveraging caches directly can significantly reduce the prefill overhead. Even simple summarization before escalation can significantly reduce prefill, but quality will be a key metric to watch closely. 

Contributions

I/ONX extends its thanks to Andrew Matthews and Jigish Shukla for their efforts on this project.