How ProsGrow Increased Qwen3 Batch Throughput by 132%

The problem: a deep queue with too few active requests

A busy request queue does not guarantee that a GPU is doing useful work at its available capacity. With Qwen3-30B-A3B on vLLM 0.25.1, ProsGrow found that the starting serving profile restricted how much of a short-prompt batch could execute together. It delivered 17,961 output tokens/s despite ample queued work.

Profiling and adjusting scheduling behavior raised that rate to 41,651 output tokens/s: a 132% improvement, or 2.32×. GPU, runtime, checkpoint, precision, request count, and prompt/output contract stayed fixed. This is a measured improvement over ProsGrow’s starting profile, not a best-tuned vLLM baseline.

Baseline → ProsGrow tuned

Median request latency also improved, from 15.91 to 11.70 seconds, in this controlled comparison. The gain identifies recoverable capacity in this workload; it does not establish a general advantage over other inference engines or fully optimized deployments.

Why the scheduler mattered

ProsGrow separated offered load from the work the engine actually admitted, then compared completed throughput and request latency under the same workload. The evidence pointed to a scheduling bottleneck: the starting profile could not turn a deep queue into enough concurrent execution. It does not isolate a kernel-level speedup.

The engineering task was to identify the limiting part of this serving stack, evaluate its effect under controlled conditions, and validate a useful operating point. Because the model fits on one GPU, independent replicas provided a practical way to test whether recovered single-GPU capacity would carry through to the full node.

The trade-off: more concurrency eventually buys waiting

A separate saturation profile reached 45,941 and 46,701 output tokens/s in two isolated runs, with median latency near 20 seconds. Pushing load beyond that operating point added only 0.91% throughput in the same sweep while median latency rose from 20.04 to 29.85 seconds. Additional queueing had stopped buying meaningful capacity.

Memory also constrained the choice: serving overhead reduced the space available for key/value cache. Short prompts left enough room for the selected profile. Longer contexts or an interactive latency target require their own evaluation.

From one tuned GPU to an eight-GPU batch service

With eight independent replicas, two synchronized runs delivered 362,812 and 361,679 output tokens/s. We use the lower result for the capacity estimate. Each run completed all 32,768 requests and 8,388,608 output tokens, with no failed or short responses.

The lower run used a common wall-clock window of 23.19 seconds, from the first client start to the last completion. Counting all work inside that shared interval avoids inflating the node result by adding independently timed replica rates.

How the 132% local gain relates to the 8.17% public comparison

The 132% gain above measures a scheduling adjustment on short queued requests. The 8.17% comparison in our capability report answers a different question: how does a local run perform against a retained public reference on a longer, specified request contract?

The public-contract run used queued synthetic requests and TensorRT-LLM 1.3.0rc24, versus 1.1 in the retained NVIDIA reference. Its mean request latency was 173.04 seconds under that deep queue. The NVIDIA reference page is the source of the 9,938 figure captured on August 17, 2026; the row was no longer exposed in our later source review. We retain it as a dated comparison, with the public performance recipe defining the workload.

The +8.17% belongs to the isolated single-GPU result. Neither it nor the difference between 85,409 and 361,679 node tokens/s isolates a software-only improvement: the latter comparison changes runtime and request lengths. These rates describe batch capacity; an interactive service needs its own latency target.

What this changes for a customer

Private generation queues and synthetic-data jobs can benefit from finishing work faster on installed capacity. ProsGrow can measure recoverable serving capacity before recommending more GPUs.

The selected throughput profile needs a deep queue and accepts roughly 20-second median latency. Deployment qualification must use the customer’s model, request-length distribution, quality checks, and traffic trace. Interactive assistants need their own latency target.

Methodology and limitations

• Hardware and software: tests ran August 18, 2026 on an eight-GPU NVIDIA RTX PRO 6000 Blackwell Server Edition node. The controlled sweep used one GPU. The runtime was vllm/vllm-openai:v0.25.1, with ModelOpt NVFP4 weights and FP8 key/value cache.
• Model and requests: the pinned NVIDIA Qwen3-30B-A3B-FP4 checkpoint used revision 2538ded2a4edb247b4d2b4a8ba24e44bd4c017c3. A repeated technical-guide prompt with a changing request identifier produced 63–66 API-reported input tokens. Temperature was zero, thinking was disabled, and every response reached the 256-token cap.
• Measurement: throughput is completed output tokens divided by client-observed elapsed time, including prompt processing and local HTTP overhead. The 132% comparison uses one measurement per profile. The selected saturation profile has two isolated repeats and two synchronized node repeats; the latter span only about 23 seconds each.
• Quality and operations: fixed output length verifies the amount of generated work, not task quality. These are local saturation measurements on an older Qwen3 checkpoint, not a sustained production soak, a customer SLA, or a quality-equivalence test. Representative prompts, p95/p99 latency, network overhead, failures, and domain quality remain production acceptance gates.

The historical vLLM 0.23 comparison is also excluded from our headline delta because its original raw prompt requests were not retained. The same-version local comparison supplies the controlled baseline.