DGX Spark benchmarking on inference performance of 14 open-weight models

The DGX Spark has 128 GB of memory that the CPU and GPU share. That’s enough to hold a 120-billion-parameter model in 4-bit precision. What that memory doesn’t tell you is how fast those models answer, and how that changes when more than one request arrives at a time.

So to benchmark the DGX Spark I ran 14 open-weight models on it, from a small Gemma model to a 120-billion-parameter Nemotron, each at six load levels between 1 and 32 concurrent requests. This article presents what I measured. Other articles in this series cover the DGX Station; this one is only about the DGX Spark.

How I benchmarked the DGX Spark

I used aiperf_tool, a small harness I built around NVIDIA’s AIPerf benchmark client and the vLLM inference server. Each model runs in its own vLLM container, and the benchmark sends requests of the same size: a 1,024-token prompt (slightly more once wrapped in each model’s chat template) and exactly 512 output tokens. The model is not allowed to stop early, so every request does the same amount of work. Each load level runs for five minutes.

The load is given as concurrent requests. Each concurrent request is a slot that sends its next request the moment the previous answer is complete, without the pauses a person makes to read and type.

The benchmark measures speed: how many tokens the system generates in total, how fast each individual answer streams, and how long it takes until an answer starts (the time to first token). It doesn’t measure answer quality, and it was run without speculative decoding.

System	NVIDIA DGX Spark (GB10), 128 GB unified memory
Models	14
Concurrent requests	1, 2, 4, 8, 16, 32 (Nemotron-3-Super: up to 16)
Runs per load level	1
Inference server	vLLM v0.29.0, 80 % of memory, FP8 KV cache
Workload	1,024 input / 512 output tokens, 5 minutes per level
Data	aiperf_dgx_spark_260917

How to read the charts in this article

The charts in this article are Pareto curves that AIPerf generates for each model. Here is nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4:

The vertical axis is the total output in tokens per second. The Spark has a single GPU, so “per GPU” is simply the total. The horizontal axis is how fast one answer streams, in tokens per second per request (AIPerf labels it “per user”). Each point is one load level, and the number next to it is the concurrency.

At 1 concurrent request, on the far right, this model streams an answer at about 70 tokens per second. Moving up and to the left, concurrency rises: at 32 concurrent requests, each answer streams at about 15 tokens per second, while the system produces about 450 in total. The line connects the load levels that no other level beats on both axes at once; a point off the line has both lower total throughput and slower answers than another load level. The axis scales adjust to each model, so check them before comparing two charts.

All 14 models that I benchmarked on the DGX Spark

Model	Architecture	Precision	Size	1 request: tok/s, TTFT	16 concurrent: tok/s per request, total	32 concurrent: tok/s per request, total	Peak total (at)
google/gemma-4-E4B-it	dense, small	BF16	15 GB	18.5, 0.26 s	20.2, 317	18.4, 569	569 (32)
google/gemma-4-E4B-it-qat-w4a16-ct	dense, small	4-bit QAT	11 GB	50.6, 0.23 s	43.1, 667	34.9, 1,075	1,075 (32)
google/gemma-4-26B-A4B-it	MoE, 4B active	BF16	48 GB	23.2, 0.41 s	9.1, 124	7.0, 199	199 (32)
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	MoE, 3B active	NVFP4	20 GB	69.9, 0.22 s	20.9, 324	14.6, 452	452 (32)
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4	MoE, 12B active	NVFP4	75 GB	15.7, 0.70 s	5.1, 80	not tested	80 (16)
ibm-granite/granite-4.2-30b	dense, 30B	BF16	55 GB	3.9, 0.92 s	3.5, 50	3.1, 50	50 (16 and 32)
Qwen/Qwen3.6-35B-A3B	MoE, 3B active	BF16	67 GB	29.8, 0.40 s	9.4, 124	6.6, 204	204 (32)
nvidia/Qwen3.6-35B-A3B-NVFP4	MoE, 3B active	NVFP4	22 GB	76.4, 0.19 s	22.8, 351	15.0, 462	462 (32)
Qwen/Qwen3.6-27B	dense, 27B	BF16	52 GB	4.4, 0.92 s	3.7, 50	3.1, 50	50 (16 and 32)
nvidia/Qwen3.6-27B-NVFP4	dense, 27B	NVFP4	20 GB	12.0, 0.95 s	7.6, 99	5.5, 149	149 (32)
Qwen/Qwen3.8-27B	dense, 27B	BF16	52 GB	4.4, 0.91 s	3.7, 50	3.1, 50	50 (16 and 32)
Qwen/Qwen3.8-27B-FP8	dense, 27B	FP8	29 GB	7.7, 0.60 s	6.2, 74	5.0, 154	154 (32)
Inferact/Qwen3.8-27B-NVFP4	dense, 27B	NVFP4	25 GB	9.5, 0.58 s	7.2, 99	5.5, 149	149 (32)
unsloth/Qwen3.8-27B-NVFP4	dense, 27B	NVFP4	22 GB	11.0, 0.52 s	7.9, 102	6.0, 149	149 (32)

Tokens per second per request is how fast one answer streams; total is the output tokens of all requests completed within the measurement window, divided by its length. TTFT is the mean time to first token. Size is the model’s repository size on Hugging Face. “Active” is the number of parameters a mixture-of-experts (MoE) model uses for each generated token, out of its total.

Below I dive into a selection of the model families I benchmarked to give you more detail.

Gemma 4

The three Gemma models cover two quite different designs. gemma-4-E4B-it is a small dense model, and the QAT variant is the same model trained to run with 4-bit weights. gemma-4-26B-A4B-it is a mixture-of-experts model with 26 billion parameters, of which about 4 billion are used per token.

The QAT variant reached the highest total throughput of all 14 models on the Spark, 1,075 tokens per second at 32 concurrent requests. A single request streamed at about 51 tokens per second, and at 16 concurrent requests each answer still streamed at about 43. The BF16 version of the same model was slower per request, between about 18 and 22 tokens per second, but held that speed almost unchanged from 1 to 32 concurrent requests while its total rose to 569.

google/gemma-4-E4B-it-qat-w4a16-ct on the DGX Spark.

One detail in this chart: the point for 1 concurrent request sits off the line, slightly to the left of the point for 2. A single request streamed a little slower (about 51 tokens per second) than each of two concurrent requests (about 55). The BF16 version shows the same pattern. With only one run per load level on the Spark, I can’t say whether that is noise or a property of the setup.

google/gemma-4-E4B-it on the DGX Spark.

gemma-4-26B-A4B-it started at about 23 tokens per second for a single request and reached 199 in total at 32 concurrent requests, with each answer streaming at about 7 tokens per second.

google/gemma-4-26B-A4B-it on the DGX Spark.

Nemotron

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, the model in the example chart above, is a mixture-of-experts model with about 3 billion active parameters, stored in NVFP4, NVIDIA’s 4-bit format for Blackwell GPUs. It streamed a single answer at about 70 tokens per second, the second highest single-request speed in this test, and reached 452 tokens per second in total at 32 concurrent requests.

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 is the largest model in this test: 120 billion parameters, 12 billion of them active per token. On the Spark, it generated about 16 tokens per second for a single request and 80 in total at 16 concurrent requests, the highest level I tested for this model.

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 on the DGX Spark. Tested up to 16 concurrent requests.

Granite

ibm-granite/granite-4.2-30b is a dense 30-billion-parameter model in BF16. Every token passes through all of its parameters, about 55 GB of weights. It generated about 4 tokens per second for a single request. AIPerf reports a total of 50 tokens per second at both 16 and 32 concurrent requests. vLLM’s own token counter, which AIPerf recorded alongside, shows generation rising from about 51 to about 96. At 32, each request took about three minutes, and AIPerf counts only requests that complete within its measurement window. The mean time to first token at 32 rose to about 14 seconds. The dense Qwen models in BF16 below show the same pattern.

ibm-granite/granite-4.2-30b on the DGX Spark. The points for 16 and 32 concurrent requests sit at about the same total throughput.

Qwen

The Qwen group has two parts: the mixture-of-experts model Qwen3.6-35B-A3B, and two dense 27-billion-parameter models, Qwen3.6-27B and Qwen3.8-27B. I ran several of them in more than one precision.

Qwen3.6-35B-A3B uses about 3 billion of its 35 billion parameters per token. In BF16, it generated about 30 tokens per second for a single request and 204 in total at 32 concurrent requests. NVIDIA’s NVFP4 version of the same model reached about 76 tokens per second for a single request, the highest single-request speed in this test, and 462 in total at 32.

Qwen/Qwen3.6-35B-A3B on the DGX Spark.

nvidia/Qwen3.6-35B-A3B-NVFP4 on the DGX Spark.

The dense 27B models in BF16 generated about 4.4 tokens per second for a single request. With FP8 weights, Qwen3.8-27B reached 7.7; the three NVFP4 versions from NVIDIA, Inferact and unsloth reached 9.5 to 12.0. At 32 concurrent requests, the quantized versions still delivered 5.0 to 6.0 tokens per second to each request, the BF16 originals 3.1.

For the BF16 versions, AIPerf’s charts show the points for 16 and 32 at the same total:

Qwen/Qwen3.8-27B (BF16) on the DGX Spark. The points for 16 and 32 concurrent requests sit at the same total throughput.

Qwen/Qwen3.6-27B on the DGX Spark. The points for 16 and 32 concurrent requests sit at about the same total throughput.

Between 16 and 32 concurrent requests, AIPerf’s total stayed at about 50 tokens per second, while vLLM’s own counter shows generation rising from about 57 to about 95. The mean time to first token rose from about 5 to about 14 seconds. The single-run timeline at 32 shows it: the 32 requests that started together got their first token between about 1 and 27 seconds after they were sent, one after another, and each then took about 2¾ minutes to generate its 512 tokens.

Each line is one request: green until its first token, orange while the answer is generated.

The FP8 and NVFP4 versions of the dense models kept adding total throughput up to 32 concurrent requests, the highest level tested; vLLM’s own counter shows 153 to 184 tokens per second at 32, against AIPerf’s 149 to 154.

nvidia/Qwen3.6-27B-NVFP4 on the DGX Spark.

Qwen/Qwen3.8-27B-FP8 on the DGX Spark.

Inferact/Qwen3.8-27B-NVFP4 on the DGX Spark.

unsloth/Qwen3.8-27B-NVFP4 on the DGX Spark.

Commonalities across the benchmarks we ran across all 14 LLM models

Two properties went along with speed on the Spark. Models that use fewer parameters per token were faster: in BF16, the mixture-of-experts models with 3 to 4 billion active parameters generated 23 to 30 tokens per second for a single request (gemma-4-26B-A4B-it 23.2, Qwen3.6-35B-A3B 29.8), the dense 27 to 30B models about 4. And lower-precision weights were faster: within each model family, the FP8, NVFP4 and 4-bit versions generated about 1.8 to 2.7 times as many tokens per second for a single request as their BF16 originals. NVIDIA’s NVFP4 version of Qwen3.6-35B-A3B ran with two additional vLLM settings (--attention-backend flashinfer, --moe-backend marlin).

At 16 concurrent requests, four models still delivered more than 20 tokens per second to each request: gemma-4-E4B-it-qat-w4a16-ct (43.1), nvidia/Qwen3.6-35B-A3B-NVFP4 (22.8), nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (20.9) and gemma-4-E4B-it (20.2).

Limits of this Spark benchmarking test

Each load level ran once on the Spark, so there is no repeat to compare against. I tested up to 32 concurrent requests, and Nemotron-3-Super up to 16. The workload is synthetic: real prompts and answers vary in length, and other settings or newer software versions can change the results. Over the whole test, one request failed, a dropped connection at 32 concurrent requests for unsloth/Qwen3.8-27B-NVFP4.

Data

All raw measurements, configuration files and charts are in aiperf_dgx_spark_260917. Each model has its own folder under artifacts/:

Model	Folder
google/gemma-4-E4B-it	20260912_223559_google_gemma-4-E4B-it_c1,2,4,8,16,32
google/gemma-4-E4B-it-qat-w4a16-ct	20260912_231103_google_gemma-4-E4B-it-qat-w4a16-ct_c1,2,4,8,16,32
google/gemma-4-26B-A4B-it	20260912_215931_google_gemma-4-26B-A4B-it_c1,2,4,8,16,32
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	20260913_014356_nvidia_NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4_c1,2,4,8,16,32
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4	20260913_011257_nvidia_NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4_c1,2,4,8,16
ibm-granite/granite-4.2-30b	20260912_234824_ibm-granite_granite-4.2-30b_c1,2,4,8,16,32
Qwen/Qwen3.6-35B-A3B	20260913_041606_Qwen_Qwen3.6-35B-A3B_c1,2,4,8,16,32
nvidia/Qwen3.6-35B-A3B-NVFP4	20260913_025945_nvidia_Qwen3.6-35B-A3B-NVFP4_c1,2,4,8,16,32
Qwen/Qwen3.6-27B	20260913_033627_Qwen_Qwen3.6-27B_c1,2,4,8,16,32
nvidia/Qwen3.6-27B-NVFP4	20260913_022142_nvidia_Qwen3.6-27B-NVFP4_c1,2,4,8,16,32
Qwen/Qwen3.8-27B	20260913_045827_Qwen_Qwen3.8-27B_c1,2,4,8,16,32
Qwen/Qwen3.8-27B-FP8	20260913_053908_Qwen_Qwen3.8-27B-FP8_c1,2,4,8,16,32
Inferact/Qwen3.8-27B-NVFP4	20260913_002824_Inferact_Qwen3.8-27B-NVFP4_c1,2,4,8,16,32
unsloth/Qwen3.8-27B-NVFP4	20260913_061847_unsloth_Qwen3.8-27B-NVFP4_c1,2,4,8,16,32

The exact vLLM settings for each model are in conf/model_compose_overrides/.

If you are evaluating the DGX Spark for a specific model or application, we would be glad to hear about your requirements. We can benchmark your model with your own prompts and answer lengths and help you assess what the results mean for your use case.