HPC Senior Systems Administrator

About the Role

KAUST (King Abdullah University of Science and Technology) is seeking a highly motivated and skilled HPC Senior Systems Administrator to join the KAUST Supercomputing Laboratory (KSL). This full-time position is based in Makkah, Saudi Arabia, and requires 5-10 years of relevant experience. The successful candidate will be responsible for managing an HPC cluster, storage systems, and networks, while providing comprehensive support to researchers and end-users across various computational domains.

Key Responsibilities

• Provide timely and effective user support via multiple channels, maintaining high customer service standards.
• Install, configure, and manage HPC subsystems, including approximately 600 CPU and GPU nodes, high-performance storage systems, InfiniBand, Ethernet, and configuration management tools (*, Ansible, Puppet).
• Deploy and manage cluster management software, monitoring tools, and supporting services for HPC clusters.
• Install and administer the Slurm workload manager, including QOS policies, accounts, accounting, and related automation scripts (Python and C++).
• Develop and maintain automation scripts in Bash and Python to streamline system administration tasks.
• Deploy and manage container environments (Singularity/Apptainer, Docker) for HPC workloads.
• Benchmark HPC system components periodically to ensure optimal performance and identify tuning opportunities.
• Enforce security best practices, including node hardening, kernel patching, and compliance across all systems.
• Manage parallel file systems such as Lustre, GPFS, Weka, or Vast, including performance tuning and capacity planning.
• Directly support research activities in computational science, engineering, data analysis, and AI/ML in collaboration with application support teams.
• Develop software tools and utilities as needed to support research projects on cluster systems.
• Drive proof-of-concept projects, technology evaluations, and advocate for system enhancements based on industry best practices.
• Coordinate with vendors and third-party service providers to resolve issues.
• Develop and maintain user documentation, standard operating procedures, and training materials.
• Stay informed on HPC advancements through continuous learning and professional collaboration, informing future hardware procurement.

Required Qualifications and Experience

• 5-10 years of experience in HPC systems administration.
• Strong expertise in Linux system administration (RHEL, Rocky Linux, or CentOS) in large-scale HPC environments.
• Demonstrated track record of managing complex HPC systems, including parallel file systems, job schedulers, InfiniBand/Ethernet networks, and monitoring systems.
• Experience with configuration management tools (Ansible, Puppet, or equivalent).
• Proficiency with HPC applications and programming models (Fortran, C/C++, Python, MPI, OpenMP, CUDA, OpenACC).
• Familiarity with computational science, data analysis, and AI/ML applications and libraries used in HPC environments.
• Knowledge of project management principles and practices.

Competencies and Skills

• Expertise in supporting users of computational science and engineering, data analysis, and artificial intelligence applications and libraries.
• Demonstrated ability to support research activities in a highly collaborative HPC environment.
• Strong analytical, problem-solving, and decision-making skills.
• Proactive in identifying and implementing system improvements, taking initiative, and seeing tasks through to closure.
• Ability to manage multiple concurrent projects and deliver high-quality results within deadlines.
• Proven ability to collaborate cross-functionally with researchers, application teams, and vendors.
• Effective in multi-cultural, international work environments.
• Excellent verbal and written communication skills in English, including the ability to prepare and deliver technical reports and presentations.