How ProsGrow Cut Raw RAG Vector Storage by 75%

The problem: retrieval has two different resource costs

A private RAG system pays to encode documents, store their vectors, and rank search results. Its useful operating point depends on which quality changes are acceptable and where capacity is constrained. ProsGrow evaluated those decisions systematically, using Qwen’s supported representations and vLLM’s precision paths.

We selected 1,024-dimensional vectors with 75% fewer raw vector bytes and, in a separate experiment, dynamic FP8 with 69.10% higher embedding throughput. Each has its own quality trade-off.

Decision 1: keep the dimensions that earn their storage

We started with Qwen3-Embedding-8B’s native 4,096-dimensional output. On the full SciFact test split, this FP16 baseline scored 0.79279 nDCG@10, a relevance measure that rewards useful documents near the top of the results. Reducing output to 1,024 dimensions scored 0.78944: 99.58% of the baseline nDCG@10.

More aggressive compression produced a larger ranking loss. We selected 1,024 dimensions for this corpus, with Recall@100 declining from 0.98000 to 0.97333. The 99.58% retention figure refers specifically to nDCG@10 in this FP16 dimension experiment.

Why smaller vectors help—and where they do not

Qwen3-Embedding-8B provides Matryoshka representations, supported by vLLM’s embedding runtime. This upstream capability supports shorter useful vectors. ProsGrow’s contribution was evaluating the resulting storage and quality trade-off for the workload.

For 100 million float32 vectors, raw storage falls from a calculated 1.64 TB to 0.41 TB. Total database footprint and search latency still require measurement with the actual index.

The encoder still computed its native hidden width, and throughput changed by at most 1.2% across the tested dimensions. The later 69.10% throughput gain comes from the separate precision experiment.

Decision 2: use FP8 for a measured increase in encoding capacity

With output dimensions and the short-input workload held constant, dynamic FP8 produced 319,951 input tokens/second across eight replicas, up from 189,213 with FP16. The lower of two FP8 node runs gives the 69.10% improvement.

The faster execution path comes from vLLM’s dynamic FP8 support. ProsGrow verified that the intended FP8 kernels executed on every GPU and evaluated precision, concurrency, and replica placement together. Increasing concurrency beyond the selected profile added little capacity while approximately doubling mean latency.

Prefix caching stayed disabled in the final comparisons so repetitive benchmark inputs could not substitute cache reuse for transformer work.

This optimized node also used 44.19% less GPU-board energy per million input tokens: 3.921 Wh versus 7.025 Wh. That is a measured GPU energy improvement, with host and facility power outside the metric.

The quality cost of FP8 is a separate decision

We repeated the full SciFact quality evaluation through a matched FP16 control and two FP8 launches. At 1,024 dimensions, the conservative FP8 result scored 0.786211 nDCG@10 versus 0.789717 for the matched FP16 control: 0.003506 lower. Recall@100 stayed at 0.97333.

The other FP8 launch scored 0.790021 nDCG@10. We report the lower score to make the acceptance decision conservative. The earlier 99.58% retention figure belongs only to the original dimension experiment; it is not a combined quality guarantee for dimension reduction and FP8.

Reranking: spend compute on candidates that can change the answer

For Qwen3-Reranker-8B, we selected 20 full documents per query. The policy study changed both candidate count and document truncation, so its effect cannot be attributed to candidate count alone. The selected policy scored 0.80482 nDCG@10, with a smaller candidate set trading compute savings against retrieval recall.

On that fixed 20-document contract, dynamic FP8 improved isolated throughput from 2.7444 to 4.6958 searches/second, 71.10% higher. The conservative eight-replica result rose from 21.6248 to 44.4738 searches/second: 77,849 to 160,106 top-20 searches/hour, or +105.66%. nDCG@10 changed from 0.804821 to 0.803368 and Recall@20 stayed at 0.94000.

The node result exceeded eight times the isolated FP8 rate in two launches; the cause remains unresolved after audit. Its capacity applies to this exact configuration. The isolated 71.10% comparison is the cleaner guide to the per-GPU precision benefit.

What this changes for a customer

For a corpus that passes the 1,024-dimensional quality gate, the vector payload shrinks by three quarters. For a short-input indexing job that also passes the FP8 gate, the measured throughput corresponds to about 52 minutes per billion input tokens, versus 88 minutes at the FP16 baseline. Those are calculated processing-time estimates at the benchmark rate.

That creates room for more frequent index refreshes. Online search needs a joint capacity plan: the full-node embedding and reranking rates were measured separately.

ProsGrow’s contribution is selecting and validating an operating point against the workload’s requirements. A customer pilot needs its own relevance judgments and complete retrieval path, including answer quality and p95 latency. The benchmark establishes component trade-offs; a production service needs joint validation.

Modeled cost for the two separate components

At full paid use, the September infrastructure model places embedding at $0.00453–$0.00646 per million input tokens and top-20 reranking at $0.03258–$0.04644 per 1,000 searches, for node purchase costs of $100,000–$150,000. The calculation uses each component’s own measured full-node rate: 319,951 input tokens/s or 44.4738 searches/s.

These scenarios assume three-year amortization, 5% annual maintenance, 720 hours/month, electricity at $0.10/kWh, a 1.30 power-usage multiplier, and 6 kW active server power. That gives $5.22–$7.44 of modeled infrastructure per active node-hour. The 6 kW input is a portfolio assumption; it is separate from the measured GPU-board energy improvement reported above. Lower demand spreads fixed cost over fewer sold units.

Each component used all eight GPUs in separate tests; their maxima cannot be combined into simultaneous service capacity. Costs exclude storage, application operations, labor, networking, service reserves, and answer generation. Cost per accepted answer requires end-to-end validation.

Methodology and limitations

• Hardware and models: eight RTX PRO 6000 Blackwell Server Edition GPUs, 96 GB nominal memory each, with one replica per GPU. Embedding used Qwen/Qwen3-Embedding-8B revision 1d8ad4ca9b3dd8059ad90a75d4983776a23d44af; reranking used Qwen/Qwen3-Reranker-8B revision 77d193c791ed757ca307ee72715aa132723da912.
• Runtime: vLLM 0.25.1; the precision comparison used float16 activations with dynamic FP8 weights and a verified FP8 kernel path. Final comparisons disabled prefix caching.
• Embedding capacity: synthetic short inputs in 64-input requests, averaging 56 model tokens each, with 1,024 returned dimensions. Each node run processed 65,536 inputs. The selected FP8 common window was 11.467 seconds; the two runs differed by 0.47% with zero failed requests. Real-document quality was evaluated separately.
• Retrieval quality: full BEIR SciFact test split: 5,183 title-plus-abstract documents, 300 queries, and official relevance judgments, evaluated with the publisher’s query instruction and exhaustive cosine search. SciFact is one scientific-domain benchmark; other corpora and approximate indexes may behave differently.
• Reranker boundary: each node run handled 2,400 searches with 20 full documents per search. Candidates were fixed from the FP16 1,024-dimensional dense baseline, so this does not measure a combined FP8 embedding-plus-reranking pipeline. We report the lower observed throughput and quality.
• Timing and deployment: warm API windows run from the earliest replica start to the latest completion. Startup and payload construction are excluded. Connectors, chunking, vector-database operations, generation, and production queueing require an end-to-end pilot. Hourly and billion-token figures are arithmetic conversions, not soak tests.
• Storage and energy: the 75% reduction is for raw vectors at identical storage precision; metadata, index structures, replication, and backups add overhead. Energy figures cover GPU boards only. Results are local comparisons of specified configurations, with no model-equivalent managed-API cost claim.