Run Model-as

As generative AI scales within the enterprise, giving developers access to multiple large language models (LLMs)—from frontier models to smaller, task-specific models—can quickly lead to infrastructure sprawl. Instead of managing dozens of isolated endpoints, platform engineering teams are moving toward a Model-as-a-Service (MaaS) approach. By using an AI gateway, you can provide a unified, secure, and easily consumable API for all your developers, which reduces GPU waste and simplifies application logic. In this guide, you will build a unified entry point for AI inference that simplifies how developers interact with multiple models. You will learn how to: - Adopt a MaaS pattern by serving both Qwen/Qwen3-0.6B and TinyLlama/TinyLlama-1.1B-Chat-v1.0 behind a single OpenAI-compatible endpoint. - Build an intelligent routing infrastructure that dynamically inspects the request payload and directs traffic based on the specified model field. - Deploy a future-ready architecture designed to scale to any number of models, which handles large models with tensor parallelism across multiple GPUs. Why this matters: One endpoint for all models means simpler client code, unified authentication, and centralized monitoring—regardless of how many models you deploy. While this guide walks through a manual deployment, these llm-d capabilities are natively integrated into Red Hat OpenShift AI. This integration reduces operational overhead and the risk of managing custom machine learning infrastructure by transforming this DIY architecture into a fully supported, enterprise-grade platform. The components This deployment builds on three open source projects: - llm-d, a platform for deploying LLM inference on Kubernetes with intelligent scheduling - Gateway API Inference Extension (GAIE), a Kubernetes SIG project that adds inference-aware routing to Gateway API - agentgateway, a Rust-based data plane with native body-based routing IThis guide uses Red Hat OpenShift as the enterprise Kubernetes distribution to demonstrate this architecture, but you can apply the core concepts to any standard Kubernetes environment. I show you how to reduce GPU underutilization and simplify application logic by routing traffic to the correct model behind a single endpoint. Understanding the LLM routing traffic flow The GAIE request flow defines a five-step process: the gateway selects a route, forwards the request to an InferencePool extension, the extension gathers metrics from the pool endpoints, selects the optimal pod, and the gateway delivers the request. This guide adds a pre-routing step before that standard flow: an AgentgatewayPolicy parses the JSON request body, extracts the model field, and sets a routing header. This header conversion enables HTTPRoute matching, which connects each model name to its InferencePool . Figure 1 shows the full request path, including the AgentgatewayPolicy pre-routing step. The following list describes the five stages in detail: - AgentgatewayPolicy parses the JSON request body using a Common Expression Language (CEL) expression, extracts the model field, and sets theX-Gateway-Base-Model-Name header. This happens in the pre-routing phase, before any route matching. - HTTPRoute matching: The gateway evaluates HTTPRoute resources, each matching on theX-Gateway-Base-Model-Name header value. A match routes the request to the correspondingInferencePool . InferencePool and EPP: Unlike a standard Kubernetes Service that does round-robin,InferencePool references an endpoint picker (EPP). The gateway calls the EPP over gRPC, and the EPP selects the optimal pod based on real-time inference metrics: KV cache utilization, queue depth, active requests, and prefix cache hit rate.- Forward to vLLM: T- the gateway sends the request directly to the selected vLLM pod on port 8000. - Response: The response flows back through the gateway to the client. Before you begin Ensure that you have the following prerequisites: - OpenShift cluster with GPU nodes oc ,helm ,helmfile , andcurl command-line interface (CLI) tools- A Hugging Face token (if pulling gated models) Clone the llm-d repository, which contains the prerequisite scripts used in this guide, and navigate to it: git clone https://github.com/llm-d/llm-d.git && cd llm-d Deploying the stack Complete the following steps to deploy the multi-model inference routing stack. Step 1: Install CRDs and agentgateway This step installs the required Custom Resource Definitions (CRDs) and the agentgateway control plane. The prerequisite scripts are located in the llm-d repository that you cloned earlier. OpenShift 4.19 and later: The Ingress Operator manages Gateway API CRDs (gateway.networking.k8s.io ) and protects them with a ValidatingAdmissionPolicy . Do not install or modify these CRDs manually. Install the GAIE CRDs, the agentgateway-specific CRDs (agentgateway.dev ), and the agentgateway control plane: # Install GAIE CRDs (InferencePool and related resources) oc apply -k https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd/?ref=v1.3.0 # Install agentgateway CRDs and control plane (does not touch gateway.networking.k8s.io) helmfile apply -f guides/prereq/gateway-provider/agentgateway.helmfile.yaml OpenShift 4.18 and earlier: Gateway API CRDs are not pre-installed. Run the full installation script: ./guides/prereq/gateway-provider/install-gateway-provider-dependencies.sh helmfile apply -f guides/prereq/gateway-provider/agentgateway.helmfile.yaml The agentgateway chart (oci://ghcr.io/kgateway-dev/charts/agentgateway v2.2.0-beta.3) installs with inferenceExtension.enabled: true into the agentgateway-system namespace. Verify the control plane is running: oc get pods -n agentgateway-system # NAME READY STATUS # agentgateway-xxxxxxxxx-xxxxx 1/1 Running Step 2: Create the gateway and SCC binding Create a Gateway resource that accepts traffic from all namespaces: oc apply -f - <<'EOF' apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: inference-gateway namespace: agentgateway-system spec: gatewayClassName: agentgateway-v2 listeners: - protocol: HTTP port: 8080 name: http allowedRoutes: namespaces: from: All EOF The gateway creates a data plane deployment to handle traffic. This deployment requires permissions beyond the default OpenShift security context constraints (SCCs). Apply an SCC binding: oc apply -f - <<'EOF' apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: inference-gateway-scc namespace: agentgateway-system roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:openshift:scc:nonroot-v2 subjects: - kind: ServiceAccount name: inference-gateway namespace: agentgateway-system EOF Verify both the control plane and data plane pods are running: oc get pods -n agentgateway-system # agentgateway-* 1/1 Running (control plane) # inference-gateway-* 1/1 Running (data plane proxy) oc get gateway -n agentgateway-system # NAME CLASS PROGRAMMED # inference-gateway agentgateway-v2 True Step 3: Configure body-based routing The AgentgatewayPolicy is where the routing logic lives. It attaches to the Gateway and runs a CEL expression that parses the JSON request body, extracts the model field, and sets a header that HTTPRoute resources can match on: oc apply -f - <<'EOF' apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayPolicy metadata: name: bbr namespace: agentgateway-system spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: inference-gateway traffic: phase: PreRouting transformation: request: set: - name: X-Gateway-Base-Model-Name value: | { "Qwen/Qwen3-0.6B": "Qwen/Qwen3-0.6B", "TinyLlama/TinyLlama-1.1B-Chat-v1.0": "TinyLlama/TinyLlama-1.1B-Chat-v1.0" }[json(request.body).model] EOF The JSON map defines which model names are recognized. The keys are what clients send in the "model" field; the values are the base model names used for routing. This same map structure also supports Low-Rank Adaptation (LoRA) adapters—to add an adapter, map its name to the base model it runs on. Step 4: Deploy InferencePools Each model gets its own InferencePool, which groups model-serving pods and provides an Endpoint Picker (EPP) for intelligent load balancing. The GAIE inferencepool chart creates the InferencePool , EPP Deployment, and an HTTPRoute with header matching. export NAMESPACE=llm-d export MODEL_SERVER=vllm export IGW_CHART_VERSION=v1.3.0 oc create namespace ${NAMESPACE} || true # Pool for Qwen/Qwen3-0.6B helm upgrade --install ${MODEL_SERVER}-qwen3-06b \ --dependency-update \ --set inferencePool.modelServers.matchLabels.app=${MODEL_SERVER}-qwen3-06b \ --set provider.name=none \ --set inferencePool.modelServerType=${MODEL_SERVER} \ --set experimentalHttpRoute.enabled=true \ --set experimentalHttpRoute.baseModel="Qwen/Qwen3-0.6B" \ --version ${IGW_CHART_VERSION} \ --namespace ${NAMESPACE} \ oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool # Pool for TinyLlama/TinyLlama-1.1B-Chat-v1.0 helm upgrade --install ${MODEL_SERVER}-tinyllama-11b \ --dependency-update \ --set inferencePool.modelServers.matchLabels.app=${MODEL_SERVER}-tinyllama-11b \ --set provider.name=none \ --set inferencePool.modelServerType=${MODEL_SERVER} \ --set experimentalHttpRoute.enabled=true \ --set experimentalHttpRoute.baseModel="TinyLlama/TinyLlama-1.1B-Chat-v1.0" \ --version ${IGW_CHART_VERSION} \ --namespace ${NAMESPACE} \ oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool Note the following details about the deployment: - The Helm release name ( ${MODEL_SERVER}-qwen3-06b ) becomes theInferencePool name. - Kubernetes resource names must be valid DNS-1035 labels—no dots allowed. Use qwen3-06b instead ofqwen3-0.6b . - The experimentalHttpRoute.enabled=true option auto-creates anHTTPRoute that matches on theX-Gateway-Base-Model-Name header and routes to theInferencePool . Verify that the resources are created: oc get inferencepool -n ${NAMESPACE} # NAME AGE # vllm-qwen3-06b ... # vllm-tinyllama-11b ... oc get httproute -n ${NAMESPACE} # NAME HOSTNAMES AGE # vllm-qwen3-06b ... # vllm-tinyllama-11b ... oc get pods -n ${NAMESPACE} | grep epp # vllm-qwen3-06b-epp-* 1/1 Running # vllm-tinyllama-11b-epp-* 1/1 Running Step 5: Deploy model servers Deploy the vLLM model servers using the llm-d-modelservice Helm chart. Each model server needs a values file that specifies the model artifact and a label matching the InferencePool selector. helm repo add llm-d-modelservice https://llm-d-incubation.github.io/llm-d-modelservice/ helm repo update If your models require authentication (for example, gated Hugging Face models), create a secret with your token: oc create secret generic llm-d-hf-token \ --from-literal="HF_TOKEN=${HF_TOKEN}" \ --namespace ${NAMESPACE} The values files are minimal; only fields without defaults need to be set. Create them before running the Helm commands: cat > ms-qwen-values.yaml <<'EOF' modelArtifacts: uri: "hf://Qwen/Qwen3-0.6B" name: "Qwen/Qwen3-0.6B" size: 10Gi # authSecretName: "llm-d-hf-token" # Uncomment if the model requires authentication labels: app: "vllm-qwen3-06b" # Must match inferencePool.modelServers.matchLabels.app routing: servicePort: 8000 accelerator: type: nvidia EOF cat > ms-llama-values.yaml <<'EOF' modelArtifacts: uri: "hf://TinyLlama/TinyLlama-1.1B-Chat-v1.0" name: "TinyLlama/TinyLlama-1.1B-Chat-v1.0" size: 10Gi # authSecretName: "llm-d-hf-token" # Uncomment if the model requires authentication labels: app: "vllm-tinyllama-11b" # Must match inferencePool.modelServers.matchLabels.app routing: servicePort: 8000 accelerator: type: nvidia EOF The critical link is modelArtifacts.labels.app . This label must exactly match the matchLabels.app in the InferencePool . This label matching is how the EPP discovers which pods belong to which pool. Deploy both model servers: helm upgrade --install ms-qwen llm-d-modelservice/llm-d-modelservice \ --namespace ${NAMESPACE} \ -f ms-qwen-values.yaml helm upgrade --install ms-llama llm-d-modelservice/llm-d-modelservice \ --namespace ${NAMESPACE} \ -f ms-llama-values.yaml Verify pods have the correct labels: oc get pods -n ${NAMESPACE} --show-labels | grep "app=" # ms-qwen-* 2/2 Running ... app=vllm-qwen3-06b,... # ms-llama-* 2/2 Running ... app=vllm-tinyllama-11b,... Step 6: Attach HTTPRoutes to the gateway The HTTPRoute resources created by the inferencepool chart live in the llm-d namespace. By default, their parentRefs do not specify the gateway's namespace. Patch the parentRefs to point to the gateway in the agentgateway-system namespace: oc patch httproute vllm-qwen3-06b -n llm-d --type='json' \ -p='[{"op":"add","path":"/spec/parentRefs/0/namespace","value":"agentgateway-system"}]' oc patch httproute vllm-tinyllama-11b -n llm-d --type='json' \ -p='[{"op":"add","path":"/spec/parentRefs/0/namespace","value":"agentgateway-system"}]' If you choose to deploy HTTPRoute and InferencePool resources in different namespaces, then you must create a ReferenceGrant in the target namespace. Verify the gateway has picked up the routes: oc get gateway inference-gateway -n agentgateway-system \ -o jsonpath='{.status.listeners[0].attachedRoutes}' # Expected: 2 Verify the deployment Now that all components are deployed, you can verify the stack: # Gateway namespace oc get pods -n agentgateway-system # agentgateway-* 1/1 Running # inference-gateway-* 1/1 Running # Model serving namespace oc get pods -n llm-d # ms-qwen-* 2/2 Running # ms-llama-* 2/2 Running # vllm-qwen3-06b-epp-* 1/1 Running # vllm-tinyllama-11b-epp-* 1/1 Running # Resources oc get inferencepool -n llm-d oc get httproute -n llm-d oc get agentgatewaypolicy -n agentgateway-system Expose the gateway with an OpenShift route Create an OpenShift route to expose the gateway externally with TLS edge termination. The haproxy.router.openshift.io/timeout annotation ensures that streaming responses, such as those using stream: true , do not time out prematurely: oc apply -f - <<'EOF' apiVersion: route.openshift.io/v1 kind: Route metadata: name: inference-gateway namespace: agentgateway-system annotations: haproxy.router.openshift.io/timeout: 300s spec: to: kind: Service name: inference-gateway port: targetPort: 8080 tls: termination: edge EOF Retrieve the Route hostname: GATEWAY_URL=$(oc get route inference-gateway -n agentgateway-system -o jsonpath='{.spec.host}') Test the routing Send requests to different models through the same endpoint: # Request routed to Qwen pool curl -s -X POST https://${GATEWAY_URL}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello Qwen"}]}' # Request routed to TinyLlama pool curl -s -X POST https://${GATEWAY_URL}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", "messages": [{"role": "user", "content": "Hello Llama"}]}' # Unknown model - no matching HTTPRoute, request fails curl -s -X POST https://${GATEWAY_URL}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "nonexistent", "messages": [{"role": "user", "content": "Should fail"}]}' The gateway routes each request to the correct model back end based solely on the model field in the request body. Common issues Use the following troubleshooting tips to resolve common errors and configuration issues encountered during the deployment process. HTTPRoutes not attaching to the gateway Verify that you patched the parentRefs namespace in Step 6. Run oc get httproute -n llm-d -o yaml and check that parentRefs[0].namespace is set to agentgateway-system . Requests timing out Check the OpenShift route timeout annotation. Streaming responses (stream: true ) require longer timeouts than the default 30 seconds. Verify that haproxy.router.openshift.io/timeout: 300s is set. Model not found errors Verify that the model names in the AgentgatewayPolicy JSON map exactly match what clients send in the "model" field. The comparison is case-sensitive. Pods not selected by EPP Confirm that the modelArtifacts.labels.app value in the model server Helm values exactly matches the inferencePool.modelServers.matchLabels.app setting. Run oc get pods -n llm-d --show-labels to verify labels. Add a new model To extend this setup to serve additional models, follow these three steps: - Add an entry to the AgentgatewayPolicy JSON map with the new model name. - Deploy a new InferencePool using the GAIE Helm chart withexperimentalHttpRoute.baseModel set to the new model name. - Deploy the model server via llm-d-modelservice withmodelArtifacts.labels.app matching theInferencePool selector. What's next You now have a multi-model inference gateway running on OpenShift. Requests to a single endpoint are automatically routed to the correct model based on the request payload. To add more models, follow the three-step process described in the previous section. The same pattern works for any number of models—each one gets its own pool, and the Gateway routes requests based on the model field. Alternative gateway providers This guide uses agentgateway, which provides native body-based routing via AgentgatewayPolicy . Other Gateway API implementations, such as Kgateway, Istio, or Google Kubernetes Engine (GKE), can also serve as the data plane. These implementations require the body-based routing external processing filter to extract the model name from the request body. For installation and configuration details, refer to the GAIE guide for serving multiple inference pools. Learn more The llm-d documentation provides additional deployment patterns and configuration options for running LLM inference on Kubernetes. The Gateway API Inference Extension site includes guides for advanced routing scenarios and alternative gateway providers. For details on the gateway data plane implementation, see the agentgateway repository. The llm-d-modelservice chart documentation covers additional model server configuration options. You can try these integrated capabilities by starting a Red Hat OpenShift AI no-cost 60-day trial.