Data Scientist QA Lead📣 Job Ad

About the Role

YO IT Consulting is seeking a Data Scientist Quality Assurance Lead for a contract, hourly role. This position focuses on overseeing quality, consistency, and trainer performance within data science AI training projects. The role is with a rapidly growing AI Data Services company that provides training data for major AI companies and foundation-model labs. Your expertise in data science quality will be crucial in ensuring that training data is analytically sound, reproducible, clearly documented, and aligned with client expectations.

Role Overview and Responsibilities

In this remote contractor position, you will be responsible for a range of quality assurance tasks related to data science AI training content. This includes:

• Reviewing AI-generated data science content and the work of trainers and QA personnel.
• Evaluating output quality against established project guidelines and rubrics.
• Providing precise, written feedback to contributors.
• Ensuring all contributors adhere to expected quality standards.
• Assessing work for statistical accuracy, data reasoning, model selection quality, code correctness, reproducibility, metric interpretation, business context awareness, clarity, formatting, instruction adherence, and alignment with project-specific rubrics.
• Identifying recurring quality issues and communicating updates to trainers and QAs.
• Supporting the onboarding process for new contributors.
• Maintaining relevant documentation.
• Assisting in activating contributors who are not working consistently.
• Performing technical reviews of AI-generated data science explanations, code snippets (Python/R/SQL), modeling workflows, statistical interpretations, dashboards, experiment designs, and step-by-step reasoning.
• Handling questions related to statistical assumptions, metrics, model selection, data leakage, validation, coding choices, reproducibility, and rubric interpretation.
• Managing trainer/QA activation by encouraging engagement, tracking follow-ups, and flagging availability issues.
• Reviewing data science outputs for potential risks such as misleading or statistically invalid conclusions.
• Contributing to process improvement by identifying quality gaps and helping to build scalable QA processes.

Qualifications and Experience

Candidates for this role should possess the following qualifications:

• A Bachelor’s, Master’s, or PhD degree in Data Science, Statistics, Computer Science, Machine Learning, Mathematics, Economics, Engineering, or a closely related quantitative field.
• A minimum of 2-5 years of professional experience in data science, analytics, machine learning, statistical modeling, experimentation, data engineering, technical review, or data science education.
• A strong understanding of statistics, probability, data cleaning, exploratory data analysis, feature engineering, supervised/unsupervised learning, model evaluation, experimentation, regression, classification, clustering, and validation methods.
• The ability to evaluate data science content against detailed rubrics and identify issues such as data leakage, flawed assumptions, incorrect metrics, weak methodology, non-reproducible code, or misleading conclusions.
• Proficiency in English, enabling clear communication and the ability to provide detailed technical feedback.
• Experience leading or supporting remote teams of trainers, annotators, analysts, data scientists, engineers, educators, or QAs is strongly preferred.
• Experience with AI training, data annotation, LLM evaluation, data science QA, or rubric-based technical review is a strong plus.

Technical Skills and Tools

Familiarity with the following tools and platforms is preferred:

• Programming languages and libraries such as Python (pandas, NumPy, scikit-learn), R, and SQL.
• Development environments and tools including Jupyter, Git, and MLflow.
• Data visualization tools like matplotlib and dashboarding platforms.
• Big data technologies such as Spark.
• Cloud platforms and general data platforms.
• Collaboration and project management tools including Discord, Google Sheets, Google Docs, trackers, dashboards, GitHub, and project management systems.

Work Arrangement and Application Process

This is a remote, contract-based hourly position. The selection process involves an AI interview, a domain-specific task, and an interview with a recruiter. While there is no immediate project assignment for this role, qualified candidates will be prioritized for future relevant opportunities and gain access to YO IT Consulting's expert network.