The infrastructure tooling market has absorbed AI assistants faster than almost any other software category. Every product now has a "Copilot" feature. Most of them are demos. A few are genuinely useful. Here's an honest assessment of where LLM assistance adds value in infrastructure operations today.
Where It Actually Works
Log Analysis at Scale
Feeding structured logs to an LLM with a prompt like "these logs are from a vSphere environment that just had a storage failure — what is the sequence of events and what caused it?" produces surprisingly good root-cause summaries. The LLM's value is pattern recognition across a large log volume — something that takes a human engineer 30 minutes takes the LLM 30 seconds.
The practical implementation is a RAG pipeline: chunk logs into semantic units, embed them, store in a vector database, and retrieve relevant chunks for the LLM prompt based on the error signatures you're investigating.
# Simplified log analysis pipeline
chunks = chunk_logs(log_file, window_seconds=60)
embeddings = embed(chunks)
relevant = vector_db.search(query="storage disconnection events", top_k=20)
summary = llm.complete(f"Analyse these vSphere log events: {relevant}")| Month | Savings |
|---|---|
| January | $250 |
| February | $80 |
| March | $420 |
Natural Language Runbooks
Converting existing runbooks into interactive LLM-assisted workflows reduces the cognitive load on on-call engineers at 3am. Instead of reading a 10-step document, the engineer describes the symptom, and the assistant walks them through the relevant steps while executing safe diagnostic commands.
The key constraint: the LLM must only execute read-only operations autonomously. Anything that changes state requires explicit human approval. This is not a technical limitation — it's a policy requirement that must be encoded in the tool architecture.
Kubernetes YAML Generation
For developers who understand what they want but don't know Kubernetes YAML, LLM assistance is genuinely valuable:
"I need a Deployment for a Python web app, 3 replicas, port 8000, with a ConfigMap for the DATABASE_URL environment variable, and a HorizontalPodAutoscaler that scales from 3 to 10 based on CPU at 70%"
The generated YAML is usually correct and requires only minor adjustments. The bottleneck is shifted from "learning YAML syntax" to "reviewing generated YAML" — the latter is faster and more appropriate for developers who aren't Kubernetes specialists.
Where It Doesn't Work Yet
Predictive capacity planning: LLMs can describe trends in historical metric data when prompted, but they don't reliably outperform traditional time-series forecasting (Prophet, SARIMA) for capacity predictions. The probabilistic nature of LLM outputs doesn't fit well with the precise numbers infrastructure planning requires.
Autonomous remediation: Every "autonomous self-healing" demo runs in a controlled environment with a specific failure mode the model was trained on. Real production failures are more creative. LLMs are excellent at assisting human decision-making; they're not reliable autonomous actors for infrastructure change management.
Configuration generation from scratch: LLMs generate plausible-looking NSX or vSAN configurations that have subtle errors — wrong VLAN IDs, incorrect MTU values, missing dependencies. These errors are dangerous precisely because the output looks correct. Always validate generated configurations against your actual environment.
The Right Architecture
The useful pattern is human-in-the-loop with LLM acceleration:
- LLM surfaces relevant context (logs, documentation, similar past incidents)
- Human engineer makes the decision
- LLM generates the command or configuration change
- Human reviews and approves
- Human executes (or approves an automation to execute)
Tools like VMware Aria Intelligence are moving in this direction — AI-assisted analysis with human-controlled remediation. That's the right model for infrastructure operations where the blast radius of a wrong decision can be significant.
The teams getting the most value from AI in infrastructure are treating LLMs as a very fast junior engineer who needs supervision, not an autonomous agent. That mental model leads to better tool design and fewer production incidents.