About the Role
Qualcomm Middle East Information Technology Company LLC is expanding its operations in Riyadh, Saudi Arabia, and is seeking a Senior Model Accuracy Development and Test Engineer for its Datacentre AI Engineering team. This role is key to supporting Qualcomm's growing infrastructure and its commitment to powering AI, cloud, and advanced connectivity at scale within the region. As Saudi Arabia progresses with its digital transformation under Vision 2030, this position offers an opportunity to contribute to a developing technology hub, working within critical environments and shaping data centre operations.
The successful candidate will focus on the design, development, and validation of model accuracy for deep learning models deployed at scale. This involves in-depth accuracy analysis, debugging, evaluation, and recovery strategies for inference on large-scale data centre hardware platforms. The role requires strong problem-solving capabilities, excellent Python programming skills, and hands-on experience with inference pipelines.
Key Responsibilities
- Define and implement accuracy Key Performance Indicators (KPIs) across various precision modes.
- Develop scalable Python-based tools and automated pipelines for accuracy evaluation.
- Implement accuracy-preserving optimizations for inference frameworks such as TensorRT, ONNX Runtime, AITemplate, and Triton.
- Build and maintain automated pipelines for accuracy evaluation across multiple frameworks including ONNX, TensorFlow, and PyTorch.
- Develop reusable plugins for preprocessing, post-processing, and metric evaluation.
- Execute comprehensive accuracy tests for large-scale models, including Large Language Models (LLMs), vision models, and diffusion models.
- Validate accuracy under various quantization and precision settings, such as FP32, FP16, and INT8.
- Perform detailed accuracy analysis with a deep understanding of model architecture, including layers, attention mechanisms, and parameter configurations.
- Identify architecture-driven accuracy degradation trends and propose effective optimization strategies.
- Identify and address issues related to preprocessing drift, tokenization mismatches, operator fallback, and quantization effects.
- Analyze accuracy differences across various hardware targets, firmware versions, and runtime backends.
- Conduct slice-based accuracy analysis considering factors like batch size, concurrency, sequence length, and domain shifts.
- Design and execute experiments to recover accuracy, which may include fine-tuning, calibration, and hyperparameter adjustments.
- Debug accuracy failures by tracing root causes across data preprocessing, model layers, quantization steps, and deployment pipelines.
- Compare results across different hardware/software stacks and generate actionable insights.
- Document workflows, maintain dashboards, and publish accuracy results for stakeholders.
Qualifications and Requirements
- Strong background in AI/ML model evaluation and accuracy metrics.
- Solid understanding of model architectures, including transformers, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Mixture-of-Experts (MoE), and their impact on accuracy.
- Experience with Large Language Models (LLMs) and generative AI accuracy validation.
- Expertise with inference runtimes such as TensorRT, ONNX Runtime, and Triton.
- Understanding of quantization techniques (INT8/FP8/INT4), calibration, Quantization-Aware Training (QAT), and associated accuracy trade-offs.
- Experience with model graph conversion processes, for example, from PyTorch to ONNX to backend engines.
- Hands-on experience with accuracy pipeline development and automation frameworks.
- Proficiency in Python and familiarity with ML toolkits like ONNX Runtime, TensorFlow, and PyTorch.
- Expertise in accuracy analysis, including the application of statistical methods and visualization tools.
- Ability to design experiments for accuracy recovery and effectively debug accuracy failures.
- Knowledge of quantization techniques and mixed-precision workflows.
- Strong problem-solving and analytical skills with the ability to isolate complex accuracy issues.
- Understanding of video generation model accuracy and multi-modal evaluation benchmarking is preferred.
- Experience with data-center accelerators such as NVIDIA A100/H100/B200, AI100 Ultra, Gaudi, or TPUs is preferred.
- Knowledge of LLM accuracy evaluation tools like lm-eval, HELM, and synthetic benchmarks is an advantage.
- Familiarity with distributed deployment systems, including Kubernetes and cloud inference services, is preferred.
- Experience in Software Engineering.
- Proficiency in programming languages such as C, C++, and Java.
Required Skills
- AI/ML model evaluation
- Accuracy metrics
- Model architectures (transformers, CNNs, RNNs, MoE)
- Large Language Models (LLMs)
- Generative AI accuracy validation
- Inference runtimes (TensorRT, ONNX Runtime, Triton)
- Quantization (INT8/FP8/INT4)
- Calibration
- Quantization-Aware Training (QAT)
- Accuracy trade-offs
- Model graph conversion (PyTorch → ONNX → backend engines)
- Accuracy pipeline development
- Automation frameworks
- Python programming
- ML toolkits (ONNX Runtime, TensorFlow, PyTorch)
- Statistical methods
- Visualization tools
- Experiment design
- Accuracy recovery
- Debugging accuracy failures
- Quantization techniques
- Mixed-precision workflows
- Problem-solving
- Analytical skills
- Video generation model accuracy (preferred)
- Multi-modal evaluation benchmarking (preferred)
- Data-center accelerators (NVIDIA A100/H100/B200, AI100 Ultra, Gaudi, TPU) (preferred)
- LLM accuracy evaluation tools (lm-eval, HELM, synthetic benchmarks) (advantage)
- Distributed deployment systems (Kubernetes, cloud inference services) (preferred)
- Software Engineering
- C programming
- C++ programming
- Java programming
Qualifications
A Bachelor's or Master's degree in Engineering, Machine Learning/AI, Information Systems, Computer Science, or a related field is required.
Minimum Qualifications:
- Bachelor's degree in Engineering, Information Systems, Computer Science, or related field and 4+ years of Software Engineering or related work experience.
- OR Master's degree in Engineering, Information Systems, Computer Science, or related field and 3+ years of Software Engineering or related work experience.
- OR PhD in Engineering, Information Systems, Computer Science, or related field and 2+ years of Software Engineering or related work experience.
- 2+ years of work experience with Programming Languages such as C, C++, Java, Python, etc.
References to a particular number of years of experience are for indicative purposes only. Applications from candidates with equivalent experience will be considered, provided that the candidate can demonstrate an ability to fulfill the principal duties of the role and possesses the required competencies.
Work Environment and Compensation
This is a full-time position based in Riyadh, Saudi Arabia. The role is within the Engineering Group, specifically Software Engineering.
Compensation includes a salary with housing and transport allowance, stock (RSUs), and a performance-related bonus. Additional benefits include 16 weeks fully paid Maternity Leave, 6 weeks fully paid Paternity Leave, an employee stock purchase scheme, child education allowance, relocation and immigration support if needed, life and medical insurance, and a Live+Well reimbursement for health and recreational membership fees.
