- Found a
reload=Truebaked into the shared base image rather than into any manifest;py-spyshowed the reloader thread consuming 769 of 1100 worker ticks. Rolled the fix out across 143 manifests in 6 environments. One customer's production contour (20 services): 7.75 to 0.78 cores. Its dev and stage contours: 113 to 42 GiB memory. Verified day-over-day against untouched control services. - Found and closed an authentication bypass on four development clusters, where the ingress IP allowlist was defeated by a client-controlled
X-Forwarded-Forheader, exposing ArgoCD, Kafka UI, EMQX and a Django admin panel to the internet. Fixed at the edge without touching cluster configs; confirmed production unaffected. - Root-caused an 8-day silent content outage on a customer-facing site: a Helm
lookupreturning nil under ArgoCD regenerated Strapi's token salt on every sync, invalidating every API token, while the frontend swallowed the resulting 401s and served an empty catalogue with nothing alerting. Reproduced it deterministically, fixed it in one line, and flagged the same chart in five other environments. - Mentors a junior DevOps engineer, writes the onboarding lab programme for trainees and juniors, and owns cluster access management for 13 engineers across four development clusters.
- Diagnosed a regulator-mandated upstream filter on the production /29 egress block, where inbound traffic worked while return traffic on outbound sessions never came back. Confirmed it on the wire with tcpdump (ICMP leaving with a correct source address, zero replies) and ruled out BGP, the default route, outbound NAT, aliases, pf and source-address selection before escalating to the provider with a documented impact list.
LLM Initiative
- Built an internal LLM-powered infrastructure analytics platform from scratch across all 12 Kubernetes clusters, ingesting and normalizing Kubernetes events and Prometheus metrics into Qdrant.
- Designed the retrieval layer (Qdrant, Nomic Embed Text v2 embeddings plus a reranker) and benchmarked llama.cpp, vLLM and Transformers before selecting Qwen 3.5 on quality against latency. Built model fallback routing so a failing or throttled provider degrades instead of stopping the pipeline (DeepSeek primary, Gemini fallback), with every stage gated by automated sanity checks on its output.
- Run the agentic tooling the team depends on day to day: self-hosted Model Context Protocol servers wired into internal systems, multi-agent workflows in Claude Code where agents decompose work and verify each other, and scheduled pipelines behind systemd timers with persistent catch-up and deduplicated state so nothing repeats between runs.
- Automated weekly anomaly reports, transforming cluster monitoring from reactive (critical alerts only) to proactive; caught a degrading production node disk before failure. Used daily by the engineering team.