AI OPS Lead

Maintain Dynatrace Davis AI alerting rules service-level problem notification routing (data-plane problems → AI Squad Slack, infra problems → Infra Squad Slack, P1 → PagerDuty), custom metric event alerting (guardrail_trigger_rate > 3× baseline, budget_burn_rate > 2× baseline) Build and maintain platform runbooks Vault sealed recovery (step-by-step unseal procedure, key shard holder contacts), node failure response (which pods need manual rescheduling, StatefulSet recovery order), Kafka broker failure (controller election verification, partition rebalance monitoring) Participate in on-call rotation incident response per runbook, escalation to Lead for P1 incidents, post-mortem documentation (5-why root cause, timeline, contributing factors, action items), SLO burn rate monitoring and alerting Operate Argo Rollouts canary deployments monitor canary health via Dynatrace AnalysisRun results, intervene on stalled rollouts, execute manual rollback when auto-rollback does not trigger within threshold, verify <2-minute rollback SLA with stopwatch measurements Manage GitLab Container Registry image retention policy configuration (keep last 10 versions per service, delete untagged images weekly), storage quota monitoring, registry mirror configuration for air-gapped deployment scenarios Kubernetes – Core RKE2 / Rancher  cluster provisioning on bare-metal, node configuration (kubelet args, containerd config, iDRAC integration for OOB management), control plane HA etcd operations (snapshot, restore, member add/remove), RKE2 upgrade procedure (worker nodes first, then control plane, one node at a time) Workload objects — Deployment, StatefulSet (stable network identity, ordered pod management), DaemonSet (OneAgent, OTel Collector — must run on every node), Job/CronJob (schema migrations, daily backup snapshots), init containers, sidecar injection (Istio, Vault Agent, gVisor) Storage — PersistentVolume / PersistentVolumeClaim lifecycle, StorageClass provisioner configuration, volume expansion, CSI driver operation (Rook-Ceph CSI, local-path provisioner for scratch storage), Velero backup and restore, snapshot CRDs Networking — Service types (ClusterIP, NodePort, LoadBalancer, ExternalName), DNS resolution (CoreDNS config tuning), Ingress vs Gateway API, EndpointSlice, network debugging (kubectl exec + curl, tcpdump via network debug pods, Hubble for Cilium) RBAC and security.