Senior Platform and EngOps Engineer - Cluster Operations

NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. Our work opens up new universes to explore, enables amazing creativity and discovery, and powers what were once science fiction inventions from artificial intelligence to autonomous cars. NVIDIA is looking for phenomenal people like you to help us accelerate the next wave of artificial intelligence.

Join our team of innovative engineers who develop and maintain software facilitating GPU communication, driving groundbreaking solutions in High Performance Computing and Deep Learning. We're looking for highly motivated EngOps and Platform Engineers to boost execution efficiency while managing and maintaining large GPU clusters interconnected via NVLink and InfiniBand.

What you will be doing:

• Develop automated tools to efficiently deploy, provision, and maintain extensive GPU clusters interconnected via NVLink and InfiniBand

• Implement modern DevOps tools to automate software updates, perform maintenance tasks, and monitor cluster availability, ensuring seamless operations.

• Take ownership of daily cluster failures and issues, troubleshooting them promptly to maintain optimal cluster availability and performance.

• Manage the rollout and rollback of cluster software and firmware updates, ensuring smooth transitions and minimal disruptions.

• Collaborate effectively with dynamic Engineering and Product Teams across multiple time zones to align cluster operations with evolving project requirements.

What we need to see:

• BS or MS in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.

• 5+ years of hands-on experience in deploying and administrating clusters, servers, switches, and related infrastructure.

• Automation expert with hands on skills in Ansible, Python and Shell Scripting.

• Deep understanding of operating systems, computer networks, and high-performance applications.

• Proven ability to work effectively with developers and test engineers across different teams and time zones.

• Proficient with Linux fundamentals.

Ways to stand out from the crowd:

• Familiarity with resource scheduling managers, preferably Slurm.

• Direct experience with industry standard alerting tools and emergency response practices.

• Hands-on experience with GPU-focused hardware and software, such as DGX systems and Compute Clusters.

• Proficiency in crafting and implementing a robust metrics collection and alerting infrastructure.

• Proficiency in designing large scale networking technologies and the associated challenges.