How ProsGrow Reached 45,978 Document Questions per Hour

The problem: equal question counts can leave unequal work

A private document service needs to finish a backlog accurately and on time. Distributing the same number of questions to every GPU sounds reasonable, until one worker receives more high-resolution pages or longer multimodal prompts. The slowest worker then determines when the batch finishes.

ProsGrow identified this hidden imbalance and evaluated a profile-guided assignment on the same node. Throughput rose from 44,608 to 45,978 document questions per hour: a 3.07% increase. The result shows how multimodal serving can leave capacity unused even when request counts look balanced.

Baseline → ProsGrow optimized

Both runs used Qwen3.6-27B BF16 on vLLM 0.25.1 across the same eight GPUs. Each request supplied a page image and question for a short answer. The complete validation workload contained 5,349 questions across 1,285 page images.

The baseline balanced by question count. ProsGrow used workload profiling to reduce the difference in work assigned to each replica, while preserving repeated-page locality.

All questions completed successfully, with 91.92% exact match. At the measured 4.16265 questions per page, throughput corresponds to 11,045 page images/hour. That conversion depends on question density.

What profiling revealed

Equal question counts concealed a 7.59% difference between the heaviest and lightest prompt workloads. Profiling made that imbalance visible; the selected assignment reduced it to within 0.017%. The final common-window throughput reached 98.61% of eight times the isolated single-GPU rate.

Balancing work also had to preserve reuse when several questions referred to the same page. Both assignments retained that locality, and the optimized run recorded a 76.06% multimodal processor-cache hit rate, including warmups. The measured gain came from distributing work more evenly on this serving foundation.

Prompt volume does not explain every straggler: page shape, batching, and request tails still affect completion. The useful signal was the gap between nominally balanced requests and the work they created, rather than a universal scheduling rule.

The trade-offs: memory headroom and precision

Large page images made memory headroom a correctness issue as well as a performance constraint. The selected serving profile admitted the entire validation set and balanced prefill utilization against latency and the margin needed for the largest pages.

We also tested a smaller mixed-NVFP4 checkpoint under the same document contract. It occupied 20.43 GiB on disk versus 51.75 GiB for BF16, but delivered 44,970 questions/hour, 2.19% below the BF16 result. Mean ANLS fell from 0.96531 to 0.96314 and exact match from 91.92% to 91.14%. We retained BF16 for this warm document-serving configuration.

Image preprocessing, the vision tower, and long multimodal prefill remained substantial costs in the quantized path. Smaller weights did not improve peak-memory headroom under the tested runtime policy. NVFP4 reduced whole-server energy per question by 2.33%, but that benefit did not offset the throughput and quality trade-off for this workload.

What this means for a private document service

The scheduling change adds approximately 1,369 completed questions per node-hour at the measured rate. For a backlog with repeated page queries, that is more useful work from the same eight GPUs. Production adoption would need a representative profiling pass or a validated cost estimator for incoming pages; this benchmark used costs learned from the same fixed corpus.

The optimized run’s median request latency was 4.521 seconds and p95 was 9.411 seconds. Free-text questions scored 0.93498 ANLS, below the overall mean. A pilot needs quality targets for the customer’s page mix and a review policy appropriate to the cost of an incorrect answer.

From measured questions to modeled cost

The cost model divides hourly infrastructure cost by the sold portion of the measured 11,045 page-equivalents/hour capacity. It is separate from the local 3.07% throughput comparison.

The model assumes three-year amortization, 5% annual maintenance, 720 hours/month, electricity at $0.10/kWh, a 1.30 power-usage multiplier for cooling and facility overhead, 7.476590 kW active server power, and 1.016 kW idle. At 10% paid demand, sold volume is 10% of benchmark capacity and the node still incurs amortization, maintenance, and idle electricity. Labor, application operations, networking, storage, and service reserves are excluded.

For price context, AWS Textract’s Queries pricing example 5 lists $15 per 1,000 pages in US West (Oregon), checked September 15, 2026. The report’s illustrative $10-per-1,000-page service price is 33.33% below that reference. It is a pricing scenario; our measured workload contains about 4.16 questions per page, and equivalent extraction quality, feature coverage, and delivered service were not tested.

Methodology and limitations

• Model and runtime: Qwen/Qwen3.6-27B, revision 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9, BF16; vLLM 0.25.1 with FlashAttention 2. Eight independent replicas on RTX PRO 6000 Blackwell Server Edition GPUs with 96 GB nominal memory each. Serving settings were held constant; thinking and prefix caching were disabled and multimodal processor caching was enabled.
• Data and scoring: the full lmms-lab-encoder/DocVQA validation split at revision 539088ef8a8ada01ac8e2e6d4e372586748a265e: 5,349 questions and 1,285 unique images. Requests asked for short answers with deterministic sampling. Quality and capacity come from the same full-split optimized run.
• Metric definition: Average Normalized Levenshtein Similarity (ANLS) measures similarity to reference answers; exact match normalizes case and whitespace. These DocVQA measures do not directly measure field-level business errors.
• Timing boundary: warm API document-question requests through completed answers, using a synchronized common node window. Hourly rates are conversions of a roughly seven-minute run. PDF ingestion, enterprise connectors, schema validation, human review, cold startup, and availability engineering are outside this measurement.
• Comparison limits: changing batch shapes changed 35 normalized predictions despite temperature zero. The small ANLS increase is observed numerical variation, not evidence that scheduling improves model accuracy. The matched scheduling comparison is a local benchmark, not a repeated production trial or a vendor-certified result. The page-rate conversion applies only to this question density.