Posts by Michael Wolf
DGX Station GB300: Qwen3.8 and Qwen3.6 in BF16, FP8 and NVFP4
Many open-weight models are published in several precisions: the original BF16 release from the model’s authors, and FP8 or 4-bit versions from the authors or from third parties. The lower-precision versions are smaller. This article looks at how eight versions of three Qwen models performed on the DGX Station GB300, from 1 to 256 concurrent…
WeiterlesenDGX Station GB300: how vLLM settings changed the results of five large models
A benchmark result describes a model running with a specific configuration, not the model on its own. I ran the same five large mixture-of-experts models twice on the DGX Station GB300, on September 17 and 18, with a few vLLM settings changed between the runs. This article compares the two runs: where the results stayed…
WeiterlesenDGX Station GB300 Benchmarking: 15 small and mid-size models up to 256 concurrent requests
The DGX Station GB300 has 288 GB of GPU memory, and each of the 15 models in this article fits into it with room to spare. For these models, the benchmarks show how many requests they process at once before each answer slows down, and at which point more load no longer adds throughput. This…
WeiterlesenDGX Spark benchmarking on inference performance of 14 open-weight models
The DGX Spark has 128 GB of memory that the CPU and GPU share. That’s enough to hold a 120-billion-parameter model in 4-bit precision. What that memory doesn’t tell you is how fast those models answer, and how that changes when more than one request arrives at a time. So to benchmark the DGX Spark…
WeiterlesenLLM inference on NVIDIA DGX Spark and DGX Station GB300 Benchmark Series
How fast a local LLM runs depends on the hardware, the model and the number of requests it processes at the same time. To measure this, I benchmarked 20 current open-weight models on an NVIDIA DGX Station GB300 and 14 of them on a DGX Spark. This series presents the results, one machine and one…
WeiterlesenWhat to expect from CRA enforcement: Learning from the MDR
Based on the provided regulatory frameworks and industry data, the Cyber Resilience Act (CRA) and the Medical Device Regulation (MDR) share a highly aligned structural architecture, despite governing different product categories. Below is a detailed comparison of their similarities, followed by documented examples and consequences related to MDR non-compliance for mid-sized and small companies. Structural…
WeiterlesenHow does LearningCircuit’s LDR (Local Deep Research) perform with Qwen 27B on an NVIDIA Spark?
Europe’s push for data sovereignty (the principle that data is subject to the laws and governance structures of the nation where it is collected) has slowly transformed from a regulatory talking point into an operational strategy. Driven by concerns over foreign surveillance laws (like the US Cloud Act) and the potential legal instability of transatlantic…
WeiterlesenOllama vs vLLM on DGX Spark For Software Engineering
Getting to run large language models like Ollama and vLLM locally on powerful developer workstation hardware like the NVIDIA DGX Spark is one of the most exciting hardware developments I’ve seen in quite a while. The DGX Spark is a very cool piece of hardware with 128gb of unified memory. Its not the fastest possible…
WeiterlesenNvidia GB300 Specs (including specifications memory bandwidth and LLM benchmarks) based on 2026 systems
The market entry this past March of Nvidia’s GB300 based DGX Station offers enterprise customers hardware with enough specs to engage with previously unavailable levels of AI performance directly within their own infrastructure. Nvidia GB300 systems like the ASUS ExpertCenter Pro ET900N G3 (which I will be testing in a coming article) bring data-center class…
Weiterlesen