For a team building a private AI assistant, GPU capacity becomes valuable when it turns into completed work. A model can generate tokens quickly and still spend substantial time handling long context, recovering earlier conversation state, or waiting between stages of an agent workflow.
That distinction shaped our GLM-5.2 study. We tested how much more output the same eight AMD MI325X GPUs could deliver by improving the way the serving system handles reusable context. The model checkpoint and precision stayed the same.
Nearly three times the output rate
In our October 6 experiment, measured output throughput increased from 3.03 to 9.05 tokens per second per GPU. Across the eight-GPU server, that was 24.22 to 72.43 output tokens per second: a 2.99× result against our local stock configuration.
| Measured result | Local stock | ProsGrow optimized |
|---|---|---|
| Output throughput per GPU | 3.03 tokens/s | 9.05 tokens/s |
| Whole-server output throughput | 24.22 tokens/s | 72.43 tokens/s |
| Median time to first token | 2.58 seconds | 1.95 seconds |
| 90th-percentile time to first token | 203.42 seconds | 10.87 seconds |
The first-token measurements show another part of the result: the long waits in this workload became substantially shorter. Median token-streaming interactivity changed by only 2.1%, so the throughput improvement should be understood as a result for the complete serving workload, rather than a claim that every token was generated three times faster.
Median full-request latency increased from 6.73 to 7.92 seconds as the completed response mix changed. Higher overall throughput therefore did not mean every latency measure improved.
Keep useful context working for longer
Agent workflows often revisit material they have already processed. Preserving that work can reduce the need to handle the same context again, especially when several conversations compete for memory.
Our approach focused on coordinating context reuse across the server’s memory resources. We used available host memory alongside GPU memory, tuned the serving policy, and checked how the configuration behaved under the full benchmark workload. Server counters showed overall cache reuse increasing from 83.4% to 96.6%.
The commercial implication is practical: a deployment assessment should examine memory and context behavior alongside GPU compute. There may be additional capacity in an existing server before another GPU purchase becomes the next step. The amount depends on the application’s traffic and response requirements.
How it compares with the published reference
Our measured 9.05 tokens/s/GPU was also 2.19× the 4.14 tokens/s/GPU in the September 25, 2026 InferenceX GLM-5.2 reference (row 443210) for eight MI325X GPUs at concurrency four. Median token-streaming interactivity was 3.6% lower than that reference. This is a comparison with a historical published point; differences in runtime identity and run conditions prevent an exact reproduction claim. Published reference run
What the result supports
Before the performance run, separate real-verification checks scored 125 out of 128 math questions under both short- and long-prompt conditions, with no baseline-correct answer becoming wrong. These checks support the tested configuration; customer workflows need their own quality evaluation.
The performance result comes from one sequential pair of one-hour AgentX tests using synthetic speculative acceptance. Some responses were shorter than requested, and the runs progressed through different response mixes with small input-length differences. All completed responses were retained. The result applies to GLM-5.2 FP8 at concurrency four; it is not a production capacity or cost guarantee.
Planning private GLM hosting or evaluating dedicated AMD GPU infrastructure? ProsGrow can help assess your model, workload and response targets—and identify which serving changes are worth testing before you expand capacity. Discuss a pilot