Operationalize AI agents with OpenShift and Kubernetes primitives

In Demystifying agentic AI: How to build production-ready AIOps with open source models, we discussed why you should use open source models for production-ready agentic AI operations (AIOps) systems. Now you need to figure out where to run them and, more importantly, how to make them operationally manageable. AI agents aren't static applications. They evolve. You'll need to: - Swap models (frontier to open source, or between open source models). - Update agent prompts and agentic routing rules. - Add new domain specialists. - Inject institutional knowledge as new failure patterns emerge. - Roll out changes without downtime. If your AI infrastructure requires rebuilding container images and redeploying every time you want to change a prompt or add a specialist, you've created an operational bottleneck. Your site reliability engineering (SRE) team won't tolerate that, and neither should you. This is part 2 of a 3-part series on building production-ready agentic AIOps systems. Part 1 explained why open source models are ready. This article shows you why Red Hat OpenShift and, specifically, how Kubernetes primitives—and specifically Red Hat OpenShift— make AI agents as operationally manageable as any other workload. The operational challenge You've deployed an agentic AIOps pipeline. It's working. But then: - You discover a gap. Package management failures go to a generic Linux specialist, but your organization has specific Red Hat Satellite content view policies the agent doesn't know about. - You need to add institutional knowledge. A new standard operating procedure (SOP) for fast-track escalations. - You want to test a different model. Switch from a 235 B parameter model to a lighter 120 B model for cost optimization. - You need to update agentic routing rules. The ops_manager should delegate package failures to a new specialist instead of the generic Linux agent. How long does this take? If the answer is "rebuild the image, push to registry, redeploy," you've lost. Kubernetes gives you operational primitives OpenShift is Kubernetes with an enterprise-grade operational layer. What makes it useful for AI workloads isn't some AI-specific feature. It's the same primitives your SRE team already uses: - ConfigMaps. Externalized configuration updates without image rebuilds. - PersistentVolumeClaims (PVCs). Durable storage for artifacts, skills, and context. - Rolling restarts. Zero-downtime updates. - Environment variables. Runtime configuration for model selection and behavior. Here's how these solve the operational challenge. Architecture: Configuration as data, not code Our agentic AIOps pipeline separates logic (container image) from configuration (ConfigMaps and PVCs): athena-agent (Deployment) ├── Container Image: quay.io/meridian/athena:1.2.3 │ └── Python code for Deep Agents harness (unchanged) │ ├── ConfigMap: athena-agent-config │ ├── AGENTS.md (ops_manager routing rules) │ └── subagents.yaml (specialist definitions: prompts, models, tools, skills) │ ├── PVC: athena-skills │ └── /app/skills/ │ ├── analyze-ansible-failure/SKILL.md │ ├── analyze-linux-failure/SKILL.md │ ├── analyze-package-management/SKILL.md (added at runtime) │ ├── analyze-openshift-failure/SKILL.md │ ├── analyze-networking-failure/SKILL.md │ ├── create-ticket/SKILL.md │ ├── error-classifier/SKILL.md │ └── common/SKILL.md │ └── Environment Variables ├── OPS_MANAGER_MODEL=qwen3-235b └── MAAS_ENDPOINT=https://maas.openshift-ai.svc.cluster.local What this means in practice: To change a model: oc set env deployment/athena OPS_MANAGER_MODEL=gpt-oss-120b - To update agentic routing rules: edit ConfigMap, oc apply , and rolling restart. - To add a new specialist: Update ConfigMap and copy skill to PVC, rolling restart. - To inject new institutional knowledge: copy markdown file to PVC. No image rebuild. No registry push. No extended downtime. Example 1: Adding a new specialist agent Let's walk through a real scenario. Our AIOps pipeline has a gap: package management failures go to sre_linux , which gives generic advice about dnf and repositories. But at our organization: - We use Red Hat Satellite with per-team content views. - Python packages for the AI/ML team require CodeReady Builder (CRB) and EPEL. - There's an SOP (v2.3) for requesting content view updates. - Fast-track escalation goes through #platform-satellite Slack channel. We need a specialist agent that knows this context. Step 1: Create the skill First, encode institutional knowledge as a markdown file: --- name: analyze-package-management-failure description: Package management failure analysis with institutional knowledge --- # Analyze Package Management Failure ## Institutional Knowledge: Satellite Infrastructure **Satellite Servers:** - Primary: satellite-primary.meridian.internal (RHEL 9 content) - Sync schedule: Tuesdays 02:00 UTC **Content View Model (per-team):** - base-rhel9: All RHEL servers, core OS packages only - ops-tooling: SRE team, monitoring, diagnostic tools - aiml-workloads: AI/ML team, Python 3.11+, CUDA, data science libs (CRB + EPEL enabled) **Content View Request Process:** - Standard: Submit per SOP v2.3 (3-5 day turnaround) - Fast-track: `#platform-satellite` Slack escalation (2-hour SLA for AI/ML workloads) ## Analysis Workflow 1. Identify the exact package(s) that failed to install from the dnf/yum error 2. Determine which team's content view is involved (check host group in AAP2) 3. Check whether the package requires CRB or EPEL 4. Reference the appropriate escalation path Copy this to the PVC: curl -sL https://gitea.example.com/configs/analyze-package-management-SKILL.md \ | oc exec --stdin deployment/athena --container athena \ -- sh -c 'cat > /app/skills/analyze-package-management/SKILL.md' The skill is now on the persistent volume, available to any agent that loads it. Step 2: Define the specialist agent Add an entry to subagents.yaml in the ConfigMap: sre_package_management: description: > Package management specialist. Delegate all package installation failures: dnf/yum errors, missing packages, Satellite content view gaps, EPEL/CRB requirements, repository sync issues, and content view request escalation. model: qwen3-235b system_prompt: | You are a senior SRE specializing in Red Hat package management and Satellite content delivery. You receive incident data from failed AAP2 jobs and perform root-cause analysis on package-related failures. Always: - Read the incident context (incident.json) first - Identify the exact package(s) that failed to install - Determine which team's content view is involved (check host group) - Reference the analyze-package-management skill for institutional knowledge - Check whether CRB or EPEL is needed and whether the team's content view includes it - Provide the SOP reference and escalation path for content view requests Use the create-ticket skill to structure your analysis as a TicketPayload. Set area to "linux" for all package management issues. tools: - web_search skills: - ./skills/analyze-package-management/ - ./skills/create-ticket/ - ./skills/common/ Download the updated config and apply it: curl -sL https://gitea.example.com/configs/subagents-with-package-mgmt.yaml \ -o /tmp/subagents-new.yaml oc create configmap athena-agent-config \ --from-file=subagents.yaml=/tmp/subagents-new.yaml \ --from-file=AGENTS.md=/tmp/agents-new.md \ --dry-run=client -o yaml | oc apply -f - Step 3: Update the error classifier The ops_manager uses an error classifier skill to identify failure domains. Update it to recognize package_management : curl -sL https://gitea.example.com/configs/error-classifier-with-package-mgmt-SKILL.md \ | oc exec --stdin deployment/athena --container athena \ -- sh -c 'cat > /app/skills/error-classifier/SKILL.md' Step 4: Rolling restart To finalize your changes and ensure the new configuration is applied, perform a rolling restart: oc rollout restart deployment/athena oc rollout status deployment/athena --timeout=120s Total time: 2 to 3 minutes. No downtime. The old pod keeps running until the new one passes health checks. The result Same infrastructure. Same error. Completely different analysis: Before (sre_linux ): Root Cause: DNF package manager reports "No package python3.14 available" Recommended Action: 1. Verify package name is correct 2. Check repository configuration 3. Consider using EPEL or enabling additional repos After (sre_package_management ): Root Cause: DNF package manager reports "No package python3.14 available" Analysis: Python 3.14 requires CRB + EPEL repositories. Host rhel-node-01 belongs to the AI/ML team content view (aiml-workloads). This content view does not currently include CRB or EPEL. Recommended Action: 1. Submit Content View Request per SOP v2.3 2. Request CRB + EPEL enablement for aiml-workloads content view 3. Escalate via #platform-satellite fast-track (2-hour SLA for AI/ML workloads) Satellite Server: satellite-primary.meridian.internal The institutional knowledge lived in a markdown file. The agent definition lived in a ConfigMap. The specialist was added without touching the container image. Example 2: Switching models at runtime You want to test whether a lighter model (gpt-oss-120b instead of qwen3-235b ) can handle specialist analysis at lower compute cost. Traditional approach: - Update Helm values. helm upgrade - Wait for image pull and pod restart. - Test. - Roll back if quality drops. The OpenShift approach: # Download updated config with new model curl -sL https://gitea.example.com/configs/subagents-gpt-oss-120b.yaml \ -o /tmp/subagents-lighter-model.yaml # Apply to ConfigMap oc create configmap athena-agent-config \ --from-file=subagents.yaml=/tmp/subagents-lighter-model.yaml \ --from-file=AGENTS.md=<(oc get configmap athena-agent-config -o jsonpath='{.data.AGENTS\.md}') \ --dry-run=client -o yaml | oc apply -f - # Rolling restart oc rollout restart deployment/athena Test the new model. If quality drops, roll back to the previous ConfigMap revision: oc rollout undo deployment/athena One command. Instant rollback. The previous pod configuration (including the original ConfigMap reference) is restored. Example 3: Changing the orchestrator model The ops_manager model is configured via an environment variable. To switch from a frontier model to open source: # Check current model oc get deployment athena -o jsonpath='{.spec.template.spec.containers[0].env}' \ | jq '.[] | select(.name == "OPS_MANAGER_MODEL")' # Switch to open source oc set env deployment/athena OPS_MANAGER_MODEL=qwen3-235b # Confirm rollout oc rollout status deployment/athena --timeout=120s That's it. No Helm. No ConfigMap editing. No image rebuild. The oc set env command updates the environment variable and triggers a rolling restart automatically. This is the same command your SRE team uses to reconfigure any application on OpenShift. AI agents aren't special. They're just workloads. Why this matters: AIOps velocity Traditional ML/AI deployment pipelines: Code Change → Build Image → Push to Registry → Update Deployment → Test → Rollback if Bad (30-60 minutes per iteration) OpenShift with externalized configuration: Update ConfigMap or PVC → Rolling Restart → Test → Rollback if Bad (2-3 minutes per iteration) When you can iterate in minutes instead of hours, you can: - Respond to new failure patterns quickly (add a specialist agent the same day). - A/B test models in production (switch, test, compare, and decide). - Continuously improve prompts (refine system prompts based on real incident data). - Inject institutional knowledge as you learn (encode new SOPs immediately). This is operational agility for AI systems. Red Hat OpenShift AI integration Red Hat OpenShift AI provides a Model-as-a-Service (MaaS) endpoint that serves open source models: ┌─────────────────────────────────────┐ │ Athena Agent (your namespace) │ │ ┌───────────────────────────────┐ │ │ │ ops_manager │ │ │ │ └─> HTTP POST to MaaS │ │ │ └───────────────────────────────┘ │ └─────────────────────────────────────┘ ↓ HTTPS ┌─────────────────────────────────────┐ │ OpenShift AI MaaS │ │ ┌───────────────────────────────┐ │ │ │ vLLM Inference Engine │ │ │ │ Models: qwen3-235b, │ │ │ │ gpt-oss-120b, │ │ │ │ llama-scout-17b, ... │ │ │ └───────────────────────────────┘ │ │ Running on Nvidia/Intel GPUs │ └─────────────────────────────────────┘ From your agent's perspective, MaaS is an endpoint. You change models by changing a string in the ConfigMap: model: qwen3-235b # 235B parameters, strongest reasoning model: gpt-oss-120b # 120B parameters, good balance model: gpt-oss-20b # 20B parameters, fast and cheap Red Hat OpenShift AI handles: - Model loading and caching. - GPU scheduling and acceleration (vLLM). - Request routing and load balancing. - Resource quotas and multi-tenancy. You handle: - Which model to use for which agent. - What configuration to load. - When to roll out changes. Observability: It's still Kubernetes Because your AI agents run as standard Kubernetes deployments, all your existing observability tooling works: # Check agent pod logs oc logs deployment/athena --tail=50 --follow # Inspect resource usage oc top pod -l app=athena # View events oc get events --field-selector involvedObject.name=athena # Exec into the pod for debugging oc exec deployment/athena -it -- /bin/bash # Check ConfigMap contents oc get configmap athena-agent-config -o yaml # List files on the PVC oc exec deployment/athena -- ls -la /app/skills/ Your SRE team doesn't need to learn "AI observability tools." They use oc logs , oc top , and oc get events (the same commands they use for every other workload). The rolling restart primitive One of the most useful patterns is the rolling restart: oc rollout restart deployment/athena What happens: - Kubernetes spins up a new pod with the updated ConfigMap/env vars. - Waits for the new pod to pass readiness checks. - Routes traffic to the new pod. - Terminates the old pod. Zero downtime. You just reconfigured an AI operations team in production without dropping a single webhook. This is what makes AI agents operationally manageable. The platform gives you: - Immutable rollouts. New config = new pod, old pod stays until new pod is ready. - Automatic rollback. oc rollout undo if something breaks. - Health checking. Bad configs don't go live if the pod can't start. - Audit trail. oc rollout history shows every deployment revision. Comparison: OpenShift versus other approaches This table highlights the operational efficiency gained by moving from traditional rebuild cycles to an externalized configuration approach on OpenShift. Approach | Config update | Model swap | Add specialist | Rollback | |---|---|---|---|---| Monolith & rebuild | Rebuild image (30 to 60 min) | Rebuild image | Rebuild image | Redeploy previous image | VMs & manual deploy | SSH + edit files + restart | Edit config + restart | Edit code + restart | Manual restore | OpenShift & externalized config | Update ConfigMap (2 to 3 min) | | ConfigMap + PVC (2 to 3 min) | | The difference isn't speed. It's confidence. When you can roll back in 30 seconds, you experiment more. When experimentation costs 1 hour and requires an image rebuild, you experiment less. Production checklist If you're running agentic AI workloads on OpenShift, here's what you should have: - ConfigMaps for agent definitions (subagent prompts, models, tools, and skills). - PVCs for skills and artifacts (markdown skills, incident data, and audit logs). - Environment variables for runtime config (model selection and API endpoints). - Rolling restart strategy (zero-downtime updates). - Rollback plan ( oc rollout undo tested and documented). - Health checks (readiness/liveness probes so bad configs don't go live). - Resource limits (CPU/memory requests and limits per agent). - Observability (logs aggregated, metrics scraped, and events monitored). These aren't AI-specific. They're Kubernetes best practices. AI agents are workloads. Next in this series This article showed you why OpenShift (how Kubernetes primitives make AI agents operationally manageable). In part 3, we'll dive into skills-driven architecture: how to extend AI agents without writing code, encode institutional knowledge in markdown, and evolve your AIOps capabilities without retraining models. Try it yourself Want hands-on experience deploying agentic AI on OpenShift with externalized configuration? Try the complete workshop: Reach out to your Red Hat representative to try the full lab. Resources - Red Hat OpenShift - Red Hat OpenShift AI - Kubernetes ConfigMaps - Kubernetes PersistentVolumeClaims - LangChain Deep Agents About this article: This post is based on a hands-on workshop created for Red Hat field demonstrations. The architecture patterns described are running in production SRE environments. Reach out to your Red Hat representative to try the full lab.