Your apps should never learn the model's name

The first question is almost always "which model should we run?"

It is a reasonable question and a bad place to start. Whatever you pick today has a shelf life measured in months. The hardware underneath it is on a three-to-five year depreciation schedule. If the answer to "which model" is baked into forty applications, you have just signed up to rewrite all forty every time the answer changes.

So we design the platform to make that question cheap to get wrong.

One address, every model

The whole architecture hangs off a single rule: every layer speaks the OpenAI-compatible API, and applications call one endpoint. They never learn a model name, a GPU count, or an IP address.

The gateway is a LiteLLM proxy behind NGINX, and it earns its place by doing five unglamorous jobs at once: routing, quotas, rate limits, audit logging, and load balancing across DGX nodes. It also holds the failover path, so when a model goes down the request lands on a backup instead of on an error page.

Above it, the access layer splits into three things that want different treatment. Employees get Open WebUI for internal chat, document Q&A, and a shared prompt library. Business applications get an OpenAI-compatible REST endpoint per team, which is a drop-in replacement for a commercial API. Identity sits alongside both, with Keycloak or whatever IdP already exists, issuing SSO sessions and group-to-model entitlements, and revoking API keys per department.

Below it, the DGX nodes run four workloads that have nothing to do with each other: a primary vLLM instance for the largest model VRAM allows, a second vLLM tuned for high-volume chat with multi-LoRA, TEI for embeddings and reranking, and LLaMA-Factory for off-peak fine-tuning runs.

The data plane stays on premises, and that is the point of the whole exercise for most of the organisations that want this. Qdrant holds document embeddings, PostgreSQL holds chat history and keys, MinIO holds source documents and model weights. Nothing leaves the building.

The payoff is narrow but real. Adding a DGX node, upgrading to a newer open-weight model, or moving quantisation from FP8 to FP4 is a gateway configuration change. It is not an application rewrite. The same design serves a single desktop unit or a rack of eight-GPU nodes.

Model-agnostic is not hardware-agnostic

Here is where the abstraction stops being free. The gateway can hide which model answers a request. It cannot invent VRAM. Somebody still has to decide how the GPUs get divided, and that decision is arithmetic, not taste.

Four GPUs at tensor-parallel 4 carry the primary general model: 437 GB of weights at FP8, leaving roughly 127 GB for KV cache. Two more at TP2 run coding and agent work. One card alone handles high-throughput chat at MXFP4, small enough that around 77 GB is left over for KV cache and one LoRA adapter per team. The last card is partitioned with MIG into seven slices, and the entire retrieval pipeline, embeddings and reranker both, fits on it with room to spare.

That last one surprises people. The RAG stack that feels architecturally significant occupies one eighth of the machine.

Now the part that decides whether any of this works: how you read a throughput number.

A published benchmark of a DGX Spark serving 256 concurrent users reports 695 tokens per second. The same run delivers 2.85 tokens per second to each user, which is slower than reading speed. Both figures are true and they are 244 times apart.

Vendor headline numbers are almost always aggregate, and they almost always assume 2:4 structured sparsity that dense inference cannot reach. Neither of those is dishonest, exactly. They are just answering a question nobody in procurement is actually asking.

So we size against per-user latency at a target concurrency, and we quote compute figures dense. It produces smaller, less impressive numbers and a system that behaves the way the spreadsheet said it would.

The boundary the abstraction cannot cross

There is exactly one place where the "it's all just configuration" story breaks, and it is worth knowing before you plan a multi-node build.

A DGX node has two networks, not one. Inside the chassis, NVLink and NVSwitch give every GPU a 1,800 GB/s path to every other GPU, 14.4 TB/s aggregate across eight of them. That fabric is soldered to the HGX baseboard. It is not a line item you choose, and it cannot leave the box. Between chassis, InfiniBand or Ethernet carries everything, at 800 GB/s for the whole node.

Drawn to scale it is an 18x drop, and that is the spec sheet being generous. Measured code agrees with the shape of it: DeepSeek's DeepEP MoE kernel moves 726 GB/s between experts inside a node and 90 GB/s between nodes, an 8x cliff in production rather than in a datasheet.

One practical rule falls out of this. Pipeline parallelism and expert parallelism can cross the node boundary. Tensor parallelism must not. A model split tensor-wise across two chassis will spend its life waiting on the slowest link in the system, and no amount of gateway configuration will fix that.

Worth noting when comparing quotes: NVIDIA markets the H200 as "10x ConnectX-7, 1 TB/s". That figure includes the two storage and management NICs, which never carry GPU collective traffic.

Eight weeks, four gates

The build order matters as much as the design, mostly because the two things that can wreck a schedule are both discoverable in week one.

Week 1 is discovery and sizing: inventory the units and VRAM, verify power, cooling and rack density, lock the priority use cases and the SLA, then re-run the sizing math on real numbers. The gate is that the design is signed off before anything is installed.

Weeks 2 and 3 build the platform foundation, and the gate is telemetry: GPU metrics and alerting verified before a single model is served. Weeks 4 and 5 bring up vLLM, the gateway, Open WebUI with SSO, and the document ingest pipeline into Qdrant, and the gate is a load test that meets the agreed SLA. Weeks 6 to 8 cover the first LoRA run, an A/B evaluation against the base model, runbooks, and operator training. That gate is the honest one: the tuned model has to beat the baseline, or it does not ship.

Two things sit outside our control and set the real start date.

Power and cooling. If the units need a rack density or a liquid loop the facility cannot supply today, siting is the long pole. Start it in Week 1, not Week 5.

Labeled data. Phase 4 cannot begin without it. If none exists, Week 1 grows to include label schema and annotation workflow design, and tuning shifts right.

Both surface in the discovery session, and neither blocks Phases 2 and 3, which deliver a working platform regardless. That last clause is what makes the schedule survivable.

What the shape actually buys

Not performance. A gateway adds a hop, and the hop costs a few milliseconds against a token stream that takes seconds.

What it buys is the ability to be wrong cheaply. Wrong about the model, and you change a config line. Wrong about quantisation, and you reload with different flags. Wrong about capacity, and you add a node behind the same address. Wrong about which department needs which entitlement, and you edit a group in the IdP instead of redeploying an application.

The alternative is not faster. It is just a system where every one of those mistakes costs a release cycle across every team that integrated.

Pick the model second. Build the seam first.

---

We design and build platform and AI infrastructure like this end to end, from sizing through handover. If you are scoping something similar and want a second pair of eyes on the topology, get in touch.