Remote AI Quality Analyst (Arabic)📣 Job Ad

About the Role

Turing, a leading research accelerator for cutting-edge AI labs and a trusted partner for global enterprises deploying advanced AI systems, announces a need for a Remote AI Quality Analyst fluent in Arabic. This contract role, focused on Saudi Arabia, is vital for evaluating a new personalization feature for Gemini. The incumbent will assess how effectively the AI model leverages user data from past conversations, Gmail, Google Search, and YouTube activity to provide more relevant and helpful responses. The role demands a unique blend of creativity in prompt engineering and analytical rigor in evaluating AI outputs.

Role Responsibilities

• Design and execute multi-turn conversational prompts, typically ranging from 1 to 5 turns, requiring the AI to use personal information and expertise.
• Evaluate model responses against the initial prompt's intent, verifying appropriate application of personalization.
• Analyze responses for grounding issues, ensuring claims about the user are supported by evidence and not based on false inferences or hallucinations.
• Assess the quality of integration to ensure personal data is seamlessly incorporated into the response without sounding robotic or overly narrative.
• Rigorously evaluate and rank two model responses side-by-side (SxS) to determine which is more helpful, user-friendly, and enjoyable overall.
• Write clear and defensible justifications for comparisons, explicitly referencing specific turns where conversational issues or positive aspects occurred.
• Extract and verify "Debug Info" from the model to confirm correct usage of chat summaries and data sources.
• Maintain strict data hygiene by deleting evaluation conversations to prevent contamination of future chat logs.

Qualifications and Requirements

• Ability to read and write Arabic at a high level of proficiency, as Arabic is the language of focus for this project.
• Willingness to use a primary personal Google account (not a test account) and enable personal data sources for genuine evaluation.
• Full-time availability in the local time zone is required, to contribute to a 24-hour global operations team.
• Proven ability to evaluate nuanced and subjective AI responses, with a particular focus on personalization quality.
• Experience in designing creative, multi-turn starter prompts based on personal context to thoroughly test model capabilities.
• Understanding of personalization concepts, including the ability to identify incorrect personalization, weak inferences, and forced connections.
• Meticulous attention to detail, with the ability to review model responses side-by-side (SxS) and identify nuances in naturalness and over-narratization.
• Superior ability to write clear, concise, and structured justifications for model rankings, explicitly referencing specific turn numbers.
• Ability to provide constructive feedback and detailed commentary.
• Excellent communication and collaboration skills.
• Self-motivated and able to work independently in a remote work environment.
• A desktop/laptop setup with a good internet connection must be available.
• Bachelor's degree or equivalent experience in a relevant field such as Politics, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field.
• Experience in data annotation, AI quality evaluation, content moderation, or a related role is highly preferred.

Core Skills

• Arabic Fluency
• Personal Account Usage
• Schedule Flexibility
• Exceptional Analytical Thinking
• Creative Prompt Engineering
• Strong Evaluation Capability
• Meticulous Attention to Detail
• Excellent Writing Skills
• Feedback Provision
• Communication and Collaboration
• Independence and Self-Motivation
• Technical Setup Readiness

Additional Details

This is a contract role with Turing. It requires a minimum commitment of 4 hours per day and 30 hours per week, with 4 hours of overlap with Pacific Standard Time (PST). Two commitment options are available: 30 hours/week or 40 hours/week. The engagement duration is 3 months. Upon application, candidates will receive an email with a login link to access the portal and complete their profile.