Self-Healing a GPU Fleet at Scale

Running thousands of GPUs means hardware fails constantly: GPUs fall off the PCIe bus, memory corrupts, nodes drop off the network. Our fleet runs on custom scheduler rather than Kubernetes, so there is no managed platform to auto-repair nodes - we built our own. In this session you will learn how we detect failures from metrics, automatically drain and remediate nodes with Ansible, and the safeguards that keep automation from making an outage worse.