How ProsGrow Increased Kimi K3 Full-Request Throughput by 50% on AMD

For an AI application that reads substantial context before returning an answer, the time spent processing the input can be as important as the speed of generation. Optimizing the complete request can therefore change which serving configuration delivers the better experience.

In our Kimi K3 study, ProsGrow increased full-request output throughput by 50.1% over our existing ATOM configuration on the same AMD hardware. The result came from serving-stack changes, with the same original model checkpoint and controlled request sizes. The experimental configuration changes numerical precision in parts of the runtime; broader quality validation remains necessary.

The result: less waiting for the complete response

Our September 22 measurements used Kimi K3 on eight AMD MI325X GPUs. Each request contained exactly 8,192 input tokens and generated 256 output tokens, with one request running at a time.

Full-request throughput divides output tokens by the time to receive the complete response, including input processing and transport. These are whole-system measurements, not per-GPU rates. The two percentages describe the same underlying improvement from different perspectives; they should not be added together.

For private AI teams, this is evidence that the serving configuration deserves evaluation alongside the hardware specification. A shorter wait for a fixed response can matter for applications that combine substantial input context with concise generated output.

A 4.41× result that shows why the workload matters

We also compared the experimental configuration with a local adapter around Microsoft's Tutel runtime, using the same host, original checkpoint, request fixtures, and client.

For the 8K-input, 256-output-token workload, the Tutel adapter delivered 9.10 full-request output tokens/s, compared with 40.17 for tuned ATOM: 4.41× the full-request rate. Median request time was 28.13 seconds for the adapter and 6.37 seconds for tuned ATOM.

Tutel still generated tokens faster once decoding was underway: 78.33 versus 47.67 tokens/s. It also completed our shorter, 512-input-token requests faster. Its longer prompt-processing time in the tested adapter changed the outcome at 8K input.

The commercial lesson is specific: choose a serving configuration against the request lengths and response sizes your application actually uses. This is a local runtime comparison, not a general claim that one runtime is faster across workloads.

Where the improvement came from

ProsGrow examined how the serving stack used AMD hardware across input processing and generation, then tested a combined runtime configuration. The work built on the ATOM and AMD AITER ecosystem.

This approach connects optimization to the whole request. It also makes the precision and quality trade-off explicit: using the same original checkpoint does not mean different serving configurations perform identical numerical operations. The runtime and cache precision policies differ in this comparison.

What we validated

Each configuration used two warmup requests and five measured requests at each input length. Speculative decoding was disabled. Saved response streams were audited to verify token counts and timings, and all three configurations passed arithmetic and marker-recall checks.

Those functional checks do not establish equivalent model accuracy. The experimental configuration remained optional rather than replacing the normal service. These measurements are a development result, without confidence intervals or a production service-level guarantee.

For teams considering private Kimi hosting, the next step is an evaluation with representative prompts, response lengths, concurrency, and task-quality criteria. ProsGrow can help connect those requirements to a measured deployment configuration.

To discuss a private inference pilot, email contact@prosgrow.ai.