DGX Station GB300: Qwen3.8 and Qwen3.6 in BF16, FP8 and NVFP4

Many open-weight models are published in several precisions: the original BF16 release from the model’s authors, and FP8 or 4-bit versions from the authors or from third parties. The lower-precision versions are smaller. This article looks at how eight versions of three Qwen models performed on the DGX Station GB300, from 1 to 256 concurrent requests.

The models are Qwen3.8-27B in four versions (BF16, FP8, and NVFP4 from Inferact and from unsloth), Qwen3.6-27B in two (BF16 and NVFP4 from NVIDIA), and Qwen3.6-35B-A3B in two (BF16 and NVFP4 from NVIDIA). The same eight measurements also appear in the article of this series that covers all 15 small and mid-size models on the Station; here they are compared with each other.

How I measured

I used aiperf_tool, a small harness I built around NVIDIA’s AIPerf benchmark client and the vLLM inference server. Each model runs in its own vLLM container, and the benchmark sends requests of the same size: a 1,024-token prompt (slightly more once wrapped in each model’s chat template) and exactly 512 output tokens. The model is not allowed to stop early, so every request does the same amount of work. Each load level runs for five minutes and was measured twice.

The load is given as concurrent requests. Each concurrent request is a slot that sends its next request the moment the previous answer is complete, without the pauses a person makes to read and type.

The benchmark measures speed: total tokens per second, how fast each individual answer streams, and the time to first token. It doesn’t measure answer quality, so it says nothing about whether a lower-precision version answers as well as the original. It was run without speculative decoding.

System	NVIDIA DGX Station GB300, 288 GB HBM3e
Models	8 versions of 3 Qwen models
Concurrent requests	1, 4, 16, 32, 64, 128; 256 for the five FP8 and NVFP4 versions
Runs per load level	2
Inference server	vLLM v0.29.0, 40 % of GPU memory, FP8 KV cache
Workload	1,024 input / 512 output tokens, 5 minutes per level
Data	aiperf_dgx_station_260918

The precisions

A model’s weights are stored as numbers, and the precision is how many bits each number uses. BF16 uses 16 bits per weight, FP8 uses 8, and NVFP4, NVIDIA’s 4-bit format for Blackwell GPUs, uses 4 plus a small amount of scaling data. The fewer bits, the smaller the model and the less data has to be read from memory for every generated token. How a model is converted to a lower precision differs between publishers.

All eight versions ran with the same vLLM version and the same memory limit of 40 % of GPU memory. A smaller model leaves more of that space for the KV cache, which holds the context of every active request. One exception: NVIDIA’s NVFP4 version of Qwen3.6-35B-A3B also ran with two additional vLLM settings for its attention and expert computation (--attention-backend flashinfer, --moe-backend marlin); the comparison with its BF16 version therefore doesn’t isolate the format.

How to read the charts

The charts in this article are Pareto curves that AIPerf generates for each model. Here is Inferact/Qwen3.8-27B-NVFP4:

The vertical axis is the total output in tokens per second. The Station has a single GPU, so “per GPU” is simply the total. The horizontal axis is how fast one answer streams, in tokens per second per request (AIPerf labels it “per user”). Each point is one load level, and the number next to it is the concurrency.

At 1 concurrent request, on the far right, this version streams an answer at about 153 tokens per second. Moving up and to the left, concurrency rises: at 256 concurrent requests, each answer streams at about 31 tokens per second, while the system produces about 7,400 in total. The line connects the load levels that no other level beats on both axes at once; a point off the line has both lower total throughput and slower answers than another load level. The axis scales adjust to each model, so check them before comparing two charts.

All eight versions

Model	Precision	Size	1 request: tok/s, TTFT	16 concurrent: tok/s per request, total, TTFT	32 concurrent: tok/s per request, total, TTFT	Peak total (at)
Qwen/Qwen3.8-27B	BF16	52 GB	89.9, 0.07 s	77.5, 1,135, 0.62 s	69.7, 1,957, 1.03 s	4,257 (128)
Qwen/Qwen3.8-27B-FP8	FP8	29 GB	111.5, 0.09 s	80.3, 940, 2.33 s	72.5, 1,414, 4.43 s	2,237 (256)
Inferact/Qwen3.8-27B-NVFP4	NVFP4	25 GB	153.2, 0.12 s	122.5, 1,783, 0.42 s	107.0, 3,005, 0.67 s	7,384 (256)
unsloth/Qwen3.8-27B-NVFP4	NVFP4	22 GB	150.0, 0.10 s	121.4, 1,768, 0.42 s	106.0, 2,973, 0.68 s	7,123 (256)
Qwen/Qwen3.6-27B	BF16	52 GB	89.9, 0.06 s	77.6, 1,137, 0.61 s	70.0, 1,955, 1.07 s	4,236 (128)
nvidia/Qwen3.6-27B-NVFP4	NVFP4	20 GB	132.4, 0.18 s	107.2, 1,263, 1.72 s	84.5, 1,787, 3.10 s	2,461 (128)
Qwen/Qwen3.6-35B-A3B	BF16	67 GB	312.4, 0.11 s	165.4, 2,459, 0.24 s	129.3, 3,742, 0.42 s	8,491 (128)
nvidia/Qwen3.6-35B-A3B-NVFP4	NVFP4	22 GB	276.7, 0.13 s	179.4, 2,549, 0.36 s	145.5, 4,021, 0.56 s	7,828 (128)

Tokens per second per request is how fast one answer streams; total is the output tokens of all requests completed within the measurement window, divided by its length. TTFT is the mean time to first token. Size is the model’s repository size on Hugging Face. The BF16 versions were tested up to 128 concurrent requests, the others up to 256.

Qwen3.8-27B

Qwen3.8-27B is a dense model: every generated token passes through all 27 billion parameters. With one request at a time, each lower-precision version was faster than the BF16 original: the FP8 version streamed at about 112 tokens per second, the two NVFP4 versions at about 150 to 153, against 90 for BF16.

Qwen/Qwen3.8-27B on the DGX Station.

Under load, the four versions separated. The two NVFP4 versions from Inferact and unsloth behaved almost identically, with about 3,000 tokens per second in total at 32 concurrent requests and 6,000 to 6,200 at 128, 42 to 45 % more than the BF16 version at that level (4,257, the highest level it was tested at). At 256, they reached 7,100 to 7,400. The FP8 version produced less in total than BF16 from 16 concurrent requests on, and peaked at 2,237 tokens per second at 256.

unsloth/Qwen3.8-27B-NVFP4 on the DGX Station. The chart for the Inferact version is the example above.

Qwen/Qwen3.8-27B-FP8 on the DGX Station.

The difference shows most clearly in the time to first token. For the FP8 version, it rose to about 2.3 seconds at 16 concurrent requests and 4.4 seconds at 32, while the BF16 version stayed at 0.6 and 1.0 seconds and the NVFP4 versions at 0.4 and 0.7. AIPerf’s chart of request throughput against time to first token shows how it develops across the load levels:

Qwen/Qwen3.8-27B-FP8 on the DGX Station: completed requests per second against the median time to first token, one point per load level.

Qwen3.6-27B

Qwen3.6-27B is also a dense 27-billion-parameter model. The BF16 versions of Qwen3.6-27B and Qwen3.8-27B produced nearly identical throughput at every load level.

Qwen/Qwen3.6-27B on the DGX Station.

NVIDIA’s NVFP4 version of Qwen3.6-27B streamed a single answer at about 132 tokens per second, faster than BF16. Under load, its total throughput rose less: 1,787 tokens per second at 32 concurrent requests and a peak of 2,461 at 128, against 1,955 and 4,236 for BF16. Its time to first token rose in a similar way to the FP8 version of Qwen3.8-27B, to about 1.7 seconds at 16 and 3.1 seconds at 32. At 256, its total throughput fell to 1,986.

nvidia/Qwen3.6-27B-NVFP4 on the DGX Station.

Qwen3.6-35B-A3B

Qwen3.6-35B-A3B is a mixture-of-experts model that uses about 3 billion of its 35 billion parameters for each token. Here the BF16 version had the higher single-request speed, about 312 tokens per second against 277 for NVIDIA’s NVFP4 version, and the higher peak, 8,491 against 7,828 at 128 concurrent requests. At 16, 32 and 64 concurrent requests, the NVFP4 version produced slightly more in total (2,549, 4,021 and 5,845 against 2,459, 3,742 and 5,713). At 256, the NVFP4 version’s total fell to 3,738; the BF16 version was not tested at that level.

Qwen/Qwen3.6-35B-A3B on the DGX Station.

nvidia/Qwen3.6-35B-A3B-NVFP4 on the DGX Station. The point for 256 sits far off the line.

Across all eight versions

For the two dense models, every lower-precision version streamed a single answer faster than its BF16 original, by 24 to 70 %. Under load, the results didn’t follow the format: at 128 concurrent requests, the highest level all versions were tested at, the Inferact and unsloth NVFP4 versions of Qwen3.8-27B produced 42 to 45 % more in total than BF16, while NVIDIA’s NVFP4 version of Qwen3.6-27B and the FP8 version of Qwen3.8-27B produced about 50 to 60 % of it, with a faster-rising time to first token. For the mixture-of-experts model, the BF16 and NVFP4 versions were within about 12 % of each other in total throughput at every level up to 128.

I didn’t investigate the causes. The KV cache was not full: vLLM’s own metrics, which AIPerf recorded alongside, show it at most 51 % full for the FP8 version and 33 % for NVIDIA’s NVFP4 version of Qwen3.6-27B at 128 concurrent requests, 84 and 66 % at 256, with no request preempted.

Limits of this test

Each load level ran twice, on one vLLM version (0.29.0) and with 40 % of GPU memory. Newer vLLM versions or newer releases of these checkpoints may change the results. The BF16 versions were not tested at 256 concurrent requests. The benchmark doesn’t compare answer quality between the precisions. The workload is synthetic, and real prompts and answers vary in length.

Data

All raw measurements, configuration files and charts are in aiperf_dgx_station_260918. Each version has its own folder under artifacts/:

Model	Folder
Qwen/Qwen3.8-27B	20260917_060722_Qwen_Qwen3.8-27B_c1,4,16,32,64,128
Qwen/Qwen3.8-27B-FP8	20260917_071251_Qwen_Qwen3.8-27B-FP8_c1,4,16,32,64,128,256
Inferact/Qwen3.8-27B-NVFP4	20260916_210841_Inferact_Qwen3.8-27B-NVFP4_c1,4,16,32,64,128,256
unsloth/Qwen3.8-27B-NVFP4	20260917_094856_unsloth_Qwen3.8-27B-NVFP4_c1,4,16,32,64,128,256
Qwen/Qwen3.6-27B	20260917_035547_Qwen_Qwen3.6-27B_c1,4,16,32,64,128
nvidia/Qwen3.6-27B-NVFP4	20260917_012119_nvidia_Qwen3.6-27B-NVFP4_c1,4,16,32,64,128,256
Qwen/Qwen3.6-35B-A3B	20260917_050205_Qwen_Qwen3.6-35B-A3B_c1,4,16,32,64,128
nvidia/Qwen3.6-35B-A3B-NVFP4	20260917_023947_nvidia_Qwen3.6-35B-A3B-NVFP4_c1,4,16,32,64,128,256

The exact vLLM settings for each version are in conf/model_compose_overrides/.

If you are deciding between precisions or checkpoints for a model on the DGX Station, we would be glad to hear about your requirements. We can benchmark the versions you are considering with your own prompts and answer lengths and help you assess what the results mean for your use case.