Jiina's Local Reasoning Model Runs at 94% of the Published DGX Spark Result

We ran Artificial Analysis's AgentPerf-Local benchmark on Jiina's local Qwen3.8-27B, took it from 26.3 to 18.5 minutes against their published 17.3, and found out why speculative decoding was doing almost nothing for agentic work.

October 17, 2026 · 7 min read

Jiina's local reasoning model now runs on one of the highest-performing setups for its hardware. We evaluated it with Artificial Analysis's open-source AA-AgentPerf-Local benchmark, a replay of real agent work that we do not control, and tuned it until it ran at 94% of Artificial Analysis's own published result for the same model on the same machine: Qwen3.8-27B on an NVIDIA DGX Spark with 128 GB of unified memory.

Behind that table is a short debugging story, and it is the interesting part. Our first clean run took 26.3 minutes. Artificial Analysis's DGX Spark finished in 17.3. This post is how we closed most of the gap, what we tried that did not help, and what the numbers say about speculative decoding on agentic work.

The benchmark, and why we trust it

AA-AgentPerf-Local is a third-party tool we do not control. It replays eight recorded agent tasks, five from PinchBench and three from SWE-bench: 168 model turns and 172 tool calls, about 2.6 million prompt tokens in total, with contexts up to 56k tokens. Each turn resends the whole conversation, and the model must produce exactly the number of tokens in the recording, so every machine does identical work. The score is total completion time, with tool time excluded. It measures speed, not quality, and prompt caching is rewarded because most prompts repeat earlier ones.

Their published figure for Qwen3.8 27B on a DGX Spark is 17.3 minutes, with llama.cpp, a 4-bit GGUF file and DFlash speculative decoding with 7 draft tokens. We serve the same model with SGLang, an NVFP4 checkpoint and speculative decoding, with thinking switched off. Same hardware, a different stack.

Where our 26 minutes went

The tool records time to first token for every turn, so we could split each run into prefill (waiting for the first token) and decode (generating the rest).

• Prefill was a flat 160 to 190 seconds in every configuration we tried. Prefix caching worked: 2.1 million of 2.36 million prompt tokens came from cache.
• Decode was everything else: about 1,390 seconds for 30,883 output tokens, or 45 milliseconds per token, roughly 15 tokens a second.

That is barely faster than plain decoding on this chip. The GB10 reads about 18.8 GB of weights for every token over 273 GB/s of memory bandwidth, which caps ordinary decoding near 12 to 14 tokens a second. Our speculative drafter was doing almost nothing.

What it was not

We chased the obvious suspects first, each on a quiet server so nothing else polluted the numbers:

• Sampling. Greedy instead of temperature 0.7: 366 seconds against 374 for 20 turns. No difference.
• Thinking mode. Turning thinking on per request: 355 seconds. No real difference.
• Long context. At 8k, 20k and 40k prompt tokens the drafter's accept length held at 5.5 to 5.9 and decoding ran near 54 tokens a second. Context length was not the problem.
• Other models on the box. The music, picture and video models were idle and used no compute.
• Server load. We checked the request log before every run to confirm nothing else was talking to the model. The runs were clean, and the clean reruns were not faster.

The real cause: the draft model, not the target

We sent the server ordinary requests and watched its speculative-decoding statistics. On Python code, our drafter, DSpark, was accepted 5.3 to 5.9 tokens per step and decoded at about 56 tokens a second. On plain prose it dropped to 1.7 to 2.1 tokens and 16 to 20 tokens a second. Agentic output, tool calls and reasoning-style text, behaves like prose.

Speculative decoding verifies a whole block of guessed tokens per step. When the guesses are right it multiplies speed. When they are wrong you still paid for the block. A drafter tuned for code looks wonderful on a coding demo and mediocre on an agent that mostly talks and plans.

We also learned how much of this benchmark is forced filler. Letting the model stop on its own for the first 20 turns produced 1,008 tokens, where the benchmark forces 5,047. A real agent finishes sooner than the benchmark makes it. It remains a fair comparison, because every machine does the same forced work, but it is worth knowing.

What worked: the model's own drafter

Qwen3.8 ships with a built-in multi-token prediction (MTP) head. Our NVFP4 checkpoint already contained it, and our SGLang build could run it with no new download. It is trained together with the model and predicts prose better.

The last row is a tie, not a win: a faster FP4 matrix-multiply backend, an automatically chosen KV-cache type and bigger prefill chunks together saved 0.1 minutes, which is inside our run-to-run noise. We left them on, but the real gains came from the drafter.

Five draft tokens was the sweet spot: four left speed on the table, six wasted work on rejected guesses. DFlash2, the drafter AA used with llama.cpp, was good, but on our stack it landed behind MTP, and it needed a newer SGLang build.

The side effect we did not expect: 25 GB back

The MTP configuration holds about 34 to 39 GB of GPU memory (34 GB right after loading, up to about 39 GB after a full benchmark run), against 60 to 63 GB for the DSpark setup. DFlash2 sat at 42 GB. With 36 GB of the box free, all six GPU models, the language model, speech recognition, text to speech, image generation, video generation and music, can now stay resident at the same time. Until now we had to move music and pictures aside whenever a video was made. We removed that.

Where we ended up

Jiina's local reasoning model now completes the benchmark in 18.5 minutes, 94% of the speed of Artificial Analysis's published DGX Spark run (17.3 minutes) and ahead of every other published system except a 32 GB RTX 5090 desktop. It does this on a different stack from theirs, SGLang with an NVFP4 checkpoint and the model's own multi-token-prediction head, while serving a 262k-token context and keeping speech recognition, text to speech, image, video and music models resident beside it. Jiina's answers are not affected: speculative decoding is lossless, so only the speed changed.

We also asked Jiina to run the benchmark herself, through her new benchmarking skill, and read the result back to us. Her full run finished in 19.8 minutes, all 168 turns passing, about 1.3 minutes behind our direct run. That is inside the run-to-run noise we saw throughout, so we quote the direct figure as the best result and her run as confirmation that the skill works end to end.

What we would tell you if you run this

1. Measure accept length per workload before you choose a drafter. A code benchmark will mislead you if your traffic is agentic.
2. Try the model's built-in MTP head first. It costs nothing to download and it may beat a third-party drafter on prose.
3. Split time into prefill and decode. They respond to completely different fixes.
4. Compare against the published stack, not just the published number. AA's 17.3 minutes comes from llama.cpp and a 4-bit GGUF file. We got within 7% of it on SGLang and NVFP4, while keeping a 262k context and the other five models resident.

Benchmark: Artificial Analysis AA-AgentPerf-Local. Hardware: one NVIDIA DGX Spark (GB10, 128 GB unified memory). All timings are single runs of the full 168-turn replay on an otherwise idle server; we saw run-to-run variation of up to 20 percent on the slow configurations, so differences under about a minute are inside the noise. We have not submitted these results to Artificial Analysis.