Forward Deployed Engineer AI Assurance📣 Job Ad

About the Role

Systems Limited is seeking a Forward Deployed Engineer AI Assurance to join their team in Saudi Arabia. This is a full-time position focused on ensuring the quality of AI-native applications. The role involves comprehensive quality assurance, including functional testing and AI-specific evaluations related to accuracy, drift, and hallucination.

Key Responsibilities

• Build test plans and automation for AI-native application features, covering both functional and AI-specific aspects.
• Design evaluation harnesses for model and agent outputs, focusing on accuracy, consistency, and hallucination rates.
• Conduct regression testing across model, prompt, and configuration changes to identify silent quality drift.
• Red-team AI features to uncover edge cases and vulnerabilities to adversarial inputs where relevant.
• Develop automated evaluation pipelines integrated into CI/CD processes.
• Collaborate with AI Architects to define testability requirements prior to development.
• Manage the quality gate for all AI features before their deployment to production.
• Communicate quality risks to delivery leadership in an actionable format.
• Provide training to delivery teams on AI-specific testing practices.
• Oversee the evaluation framework for the practice, including golden datasets, scoring rubrics, LLM-as-judge calibration, and versioned benchmarks per use case.
• Define and enforce evaluation acceptance thresholds for each engagement and gate releases based on these thresholds.
• Develop evaluation engineering tooling for dataset curation, trace capture, offline/online evaluation runs, and dashboards for delivery teams.
• Instrument production evaluations and drift monitoring, feeding identified failures back into the golden datasets.

Experience Required

Candidates should possess 5-10 years of relevant experience in quality assurance, with a specific focus on AI-native applications and AI-specific evaluation methodologies.

Work Type and Location

This is a full-time position based in Saudi Arabia. The salary for this role will be discussed during the interview process.