DGX Station GB300 Benchmarking: 15 small and mid-size models up to 256 concurrent requests

The DGX Station GB300 has 288 GB of GPU memory, and each of the 15 models in this article fits into it with room to spare. For these models, the benchmarks show how many requests they process at once before each answer slows down, and at which point more load no longer adds throughput.

This article covers 15 models between 11 and 126 GB, from small Gemma models to a 120-billion-parameter Nemotron and a 126 GB Qwen mixture-of-experts model, each measured at up to seven load levels between 1 and 256 concurrent requests. Another article in this series covers the five largest models on the same machine.

How I measured these DGX Station benchmarks

I used aiperf_tool, a small harness I built around NVIDIA’s AIPerf benchmark client and the vLLM inference server. Each model runs in its own vLLM container, and the benchmark sends requests of the same size: a 1,024-token prompt (slightly more once wrapped in each model’s chat template) and exactly 512 output tokens. The model is not allowed to stop early, so every request does the same amount of work. Each load level runs for five minutes and was measured twice.

The load is given as concurrent requests. Each concurrent request is a slot that sends its next request the moment the previous answer is complete, without the pauses a person makes to read and type.

The benchmark measures speed: total tokens per second, how fast each individual answer streams, and the time to first token. It doesn’t measure answer quality, and it was run without speculative decoding.

System	NVIDIA DGX Station GB300, 288 GB HBM3e
Models	15
Concurrent requests	1, 4, 16, 32, 64, 128, and 256 for 9 of the models
Runs per load level	2
Inference server	vLLM v0.29.0 (RadixArk Flash-Next: nightly build); 40 % of GPU memory for all models (for RadixArk Flash-Next, the configuration file in the repository says 80 %; vLLM’s own record shows 40 %, the setting at the time its container was created); FP8 KV cache, except RadixArk Flash-Next (vLLM default)
Workload	1,024 input / 512 output tokens, 5 minutes per level
Data	aiperf_dgx_station_260918

All models were limited to 40 % of the GPU’s memory for the weights and the KV cache, which holds the context of every active request.

How to read the charts in these benchmarks

The charts in this article are Pareto curves that AIPerf generates for each model. Here is nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4:

The vertical axis is the total output in tokens per second. The Station has a single GPU, so “per GPU” is simply the total. The horizontal axis is how fast one answer streams, in tokens per second per request (AIPerf labels it “per user”). Each point is one load level, and the number next to it is the concurrency.

At 1 concurrent request, on the far right, this model streams an answer at about 384 tokens per second. Moving up and to the left, concurrency rises: at 256 concurrent requests, each answer streams at about 44 tokens per second, while the system produces about 9,100 in total. The line connects the load levels that no other level beats on both axes at once; a point off the line has both lower total throughput and slower answers than another load level. The axis scales adjust to each model, so check them before comparing two charts.

All 15 models benchmarked on the DGX Station

Model	Architecture	Precision	Size	1 request: tok/s, TTFT	16 concurrent: tok/s per request, total	32 concurrent: tok/s per request, total	Highest level tested: tok/s per request, total	Peak total (at)
google/gemma-4-E4B-it	dense, small	BF16	15 GB	235.8, 0.03 s	235.6, 3,622	223.6, 6,758	256: 6.8*, 2,032	17,544 (128)
google/gemma-4-E4B-it-qat-w4a16-ct	dense, small	4-bit QAT	11 GB	195.1, 0.03 s	185.6, 2,873	164.2, 5,038	256: 7.7*, 1,932	11,825 (128)
google/gemma-4-26B-A4B-it	MoE, 4B active	BF16	48 GB	242.5, 0.04 s	140.9, 2,203	104.3, 3,253	128: 59.6, 7,217	7,217 (128)
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	MoE, 3B active	NVFP4	20 GB	383.8, 0.10 s	221.7, 3,048	157.7, 4,331	256: 44.6, 9,136	9,136 (256)
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4	MoE, 12B active	NVFP4	75 GB	213.8, 0.15 s	104.2, 1,551	77.9, 2,323	128: 37.3, 4,428	4,428 (128)
ibm-granite/granite-4.2-30b	dense, 30B	BF16	55 GB	81.6, 0.05 s	83.6, 1,314	74.1, 2,330	128: 58.9, 7,260	7,260 (128)
Qwen/Qwen3.6-35B-A3B	MoE, 3B active	BF16	67 GB	312.4, 0.11 s	165.4, 2,459	129.3, 3,742	128: 75.2, 8,491	8,491 (128)
nvidia/Qwen3.6-35B-A3B-NVFP4	MoE, 3B active	NVFP4	22 GB	276.7, 0.13 s	179.4, 2,549	145.5, 4,021	256: 15.7, 3,738	7,828 (128)
RadixArk/Qwen3.8-Flash-Next-NVFP4	MoE, 10 of 512 experts	NVFP4	126 GB	198.9, 0.14 s	114.7, 1,694	93.3, 2,694	256: 49.3, 2,605	2,694 (32)
Qwen/Qwen3.6-27B	dense, 27B	BF16	52 GB	89.9, 0.06 s	77.6, 1,137	70.0, 1,955	128: 41.7, 4,236	4,236 (128)
nvidia/Qwen3.6-27B-NVFP4	dense, 27B	NVFP4	20 GB	132.4, 0.18 s	107.2, 1,263	84.5, 1,787	256: 12.1, 1,986	2,461 (128)
Qwen/Qwen3.8-27B	dense, 27B	BF16	52 GB	89.9, 0.07 s	77.5, 1,135	69.7, 1,957	128: 37.3, 4,257	4,257 (128)
Qwen/Qwen3.8-27B-FP8	dense, 27B	FP8	29 GB	111.5, 0.09 s	80.3, 940	72.5, 1,414	256: 12.4, 2,237	2,237 (256)
Inferact/Qwen3.8-27B-NVFP4	dense, 27B	NVFP4	25 GB	153.2, 0.12 s	122.5, 1,783	107.0, 3,005	256: 31.6, 7,384	7,384 (256)
unsloth/Qwen3.8-27B-NVFP4	dense, 27B	NVFP4	22 GB	150.0, 0.10 s	121.4, 1,768	106.0, 2,973	256: 30.3, 7,123	7,123 (256)

Tokens per second per request is how fast one answer streams (mean over all requests); total is the output tokens of all requests completed within the measurement window, divided by its length. TTFT is the mean time to first token. * Median instead of mean: at 256 concurrent requests, 5 to 10 % of requests with extreme values put the mean for the two Gemma E4B variants at 76.5 and 42.1, far from what most requests saw. Size is the model’s repository size on Hugging Face. “Active” is the number of parameters a mixture-of-experts (MoE) model uses for each generated token, out of its total.

Ten of the 15 models were still gaining at the highest level I tested. The other five reached their highest total throughput earlier; the sections below show where.

Gemma 4 on the DGX Station

gemma-4-E4B-it is a small dense model, and the QAT variant is the same model trained to run with 4-bit weights. gemma-4-26B-A4B-it is a mixture-of-experts model with about 4 billion of its 26 billion parameters active per token.

gemma-4-E4B-it reached the highest total throughput of all 15 models: 17,544 tokens per second at 128 concurrent requests. Up to 16 concurrent requests, each answer streamed at about 236 tokens per second, the same speed as a single request, while the total rose about sixteenfold. At 128, each answer still streamed at about 156.

google/gemma-4-E4B-it on the DGX Station. The point for 256 sits far off the line.

At 256 concurrent requests, both Gemma E4B variants behaved very differently. Total throughput fell to about 2,000 tokens per second. For gemma-4-E4B-it, the median request streamed at about 7 tokens per second and took about 75 seconds from start to finish, against 156 tokens per second and 3.7 seconds at 128. The time to first token also varied widely, as the single-run chart shows:

Each dot is one request, in the order it was sent.

Most requests got their first token within about 7 seconds, but a group of them waited up to about 60 seconds. I didn’t investigate the cause. The KV cache was not full: vLLM’s own metrics, which AIPerf recorded alongside, show it at most 6 % full at this level (5 % for the QAT variant), and no request was preempted.

google/gemma-4-E4B-it-qat-w4a16-ct on the DGX Station. The point for 256 sits far off the line, as for the BF16 version.

gemma-4-26B-A4B-it streamed a single answer at about 243 tokens per second and reached 7,217 in total at 128 concurrent requests, its highest tested level.

google/gemma-4-26B-A4B-it on the DGX Station.

Nemotron on the DGX Station

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, the model in the example chart above, uses about 3 billion active parameters and NVIDIA’s 4-bit NVFP4 format. It had the highest single-request speed of all 15 models, about 384 tokens per second, and its total throughput was still rising at 256 concurrent requests, at 9,136 tokens per second. A request took about 14 seconds at that level.

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 has 120 billion parameters, 12 billion of them active per token. It streamed a single answer at about 214 tokens per second and reached 4,428 in total at 128, the highest level I tested for it.

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 on the DGX Station.

Granite on the DGX Station

ibm-granite/granite-4.2-30b is a dense 30-billion-parameter model in BF16. It streamed a single answer at about 82 tokens per second and held that speed up to 16 concurrent requests (about 84); beyond that it fell, to 59 at 128. Total throughput rose steadily with load, to 7,260 tokens per second at 128, its highest tested level, with a mean time to first token of 0.33 seconds.

ibm-granite/granite-4.2-30b on the DGX Station. The points for 1 and 4 sit off the line, because 16 concurrent requests streamed at least as fast per request, with a higher total.

Qwen on the DGX Station

The Qwen group includes three mixture-of-experts models and six dense 27B models in different precisions.

Qwen/Qwen3.6-35B-A3B in BF16 uses about 3 billion of its 35 billion parameters per token. It streamed a single answer at about 312 tokens per second and reached 8,491 in total at 128, its highest tested level. NVIDIA’s NVFP4 version reached 7,828 at 128; at 256, its total fell to 3,738 and a request took about 35 seconds. The NVFP4 version ran with two additional vLLM settings (--attention-backend flashinfer, --moe-backend marlin).

Qwen/Qwen3.6-35B-A3B on the DGX Station.

nvidia/Qwen3.6-35B-A3B-NVFP4 on the DGX Station. The point for 256 sits far off the line.

RadixArk/Qwen3.8-Flash-Next-NVFP4 is a larger mixture-of-experts model, 126 GB with 512 experts of which 10 are used per token. It ran on the vLLM nightly build, with the same 40 % of GPU memory as the other models. It reached its highest total, 2,694 tokens per second, at 32 concurrent requests. Beyond that, the total stayed between about 2,600 and 2,700. vLLM’s own metrics show what coincided with it: from 64 concurrent requests on, the KV cache was full and at most 56 requests ran at the same time, and vLLM preempted running requests (paused them and restarted them later) about 700 to 850 times per five-minute run. The other requests waited: the mean time to first token rose from 0.6 seconds at 32 to about 14 seconds at 128 and 36 seconds at 256.

RadixArk/Qwen3.8-Flash-Next-NVFP4 on the DGX Station. The points for 64, 128 and 256 sit at about the same total throughput as 32.

The dense 27B models, Qwen3.6-27B and Qwen3.8-27B, ran in BF16 and in FP8 and NVFP4 versions from different publishers. In BF16, both streamed a single answer at about 90 tokens per second and reached about 4,250 in total at 128. The quantized versions differ widely from each other under load, from 2,237 to 7,384 tokens per second at their peaks. Another article in this series looks at these variants in detail.

Qwen/Qwen3.6-27B on the DGX Station.

Qwen/Qwen3.8-27B on the DGX Station.

nvidia/Qwen3.6-27B-NVFP4 on the DGX Station.

Qwen/Qwen3.8-27B-FP8 on the DGX Station.

Inferact/Qwen3.8-27B-NVFP4 on the DGX Station.

unsloth/Qwen3.8-27B-NVFP4 on the DGX Station.

Across all 15 models

Single-request speed ranged from about 82 to 384 tokens per second, and peak total throughput from 2,237 to 17,544. The lowest single-request speeds, 82 to 90 tokens per second, came from the three dense 27 to 30B models in BF16; the highest from models with 3 to 4 billion active parameters.

Five models peaked before the highest load level: both Gemma E4B variants and NVIDIA’s NVFP4 versions of Qwen3.6-27B and Qwen3.6-35B-A3B at 128, and RadixArk Flash-Next at 32. For these, going beyond the peak meant lower total throughput or much longer waits.

Limits of this test

Each load level ran twice. All models were limited to 40 % of GPU memory; two of them filled their KV cache, Nemotron-3-Super at 128 concurrent requests and RadixArk Flash-Next from 64 on. RadixArk Flash-Next ran with a different KV cache format and vLLM version than the others. Six models were tested up to 128 rather than 256 concurrent requests. The workload is synthetic, and real prompts and answers vary in length.

Data

All raw measurements, configuration files and charts are in aiperf_dgx_station_260918. Each model has its own folder under artifacts/:

Model	Folder
google/gemma-4-E4B-it	20260916_171715_google_gemma-4-E4B-it_c1,4,16,32,64,128,256
google/gemma-4-E4B-it-qat-w4a16-ct	20260916_183753_google_gemma-4-E4B-it-qat-w4a16-ct_c1,4,16,32,64,128,256
google/gemma-4-26B-A4B-it	20260916_160706_google_gemma-4-26B-A4B-it_c1,4,16,32,64,128
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	20260916_234041_nvidia_NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4_c1,4,16,32,64,128,256
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4	20260916_223047_nvidia_NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4_c1,4,16,32,64,128
ibm-granite/granite-4.2-30b	20260916_195805_ibm-granite_granite-4.2-30b_c1,4,16,32,64,128
Qwen/Qwen3.6-35B-A3B	20260917_050205_Qwen_Qwen3.6-35B-A3B_c1,4,16,32,64,128
nvidia/Qwen3.6-35B-A3B-NVFP4	20260917_023947_nvidia_Qwen3.6-35B-A3B-NVFP4_c1,4,16,32,64,128,256
RadixArk/Qwen3.8-Flash-Next-NVFP4	20260917_083130_RadixArk_Qwen3.8-Flash-Next-NVFP4_c1,4,16,32,64,128,256
Qwen/Qwen3.6-27B	20260917_035547_Qwen_Qwen3.6-27B_c1,4,16,32,64,128
nvidia/Qwen3.6-27B-NVFP4	20260917_012119_nvidia_Qwen3.6-27B-NVFP4_c1,4,16,32,64,128,256
Qwen/Qwen3.8-27B	20260917_060722_Qwen_Qwen3.8-27B_c1,4,16,32,64,128
Qwen/Qwen3.8-27B-FP8	20260917_071251_Qwen_Qwen3.8-27B-FP8_c1,4,16,32,64,128,256
Inferact/Qwen3.8-27B-NVFP4	20260916_210841_Inferact_Qwen3.8-27B-NVFP4_c1,4,16,32,64,128,256
unsloth/Qwen3.8-27B-NVFP4	20260917_094856_unsloth_Qwen3.8-27B-NVFP4_c1,4,16,32,64,128,256

The exact vLLM settings for each model are in conf/model_compose_overrides/.

If you are evaluating the DGX Station for a specific model or application, we would be glad to hear about your requirements. We can benchmark your model with your own prompts and answer lengths and help you assess what the results mean for your use case.