Python(Machine leaning) QA Lead📣 Job Ad

About the Role

YO IT Consulting is seeking a Python (Machine Learning) Quality Assurance Lead for a contract role. This position will oversee quality, consistency, and trainer performance for Python machine learning AI training projects within a rapidly growing AI Data Services company that partners with major AI companies and foundation-model labs. The role requires a focus on ensuring training data is accurate, executable, statistically sound, reproducible, clearly explained, and aligned with client expectations.

Key Responsibilities

The Python (Machine Learning) Quality Assurance Lead will be responsible for the following:

• Reviewing AI-generated Python code, ML workflows, model explanations, and trainer/QA work against project guidelines.
• Evaluating output quality and providing precise written feedback to ensure contributors adhere to expected standards.
• Assessing work for correctness in code, machine learning methodology, statistical validity, reproducibility, model evaluation, data leakage risks, package usage, debugging, readability, maintainability, formatting, and instruction adherence.
• Spot-checking Python ML items, identifying quality issues, providing feedback, and escalating critical or recurring problems.
• Evaluating AI-generated Python code, ML pipelines, data preprocessing, model training, evaluation logic, debugging responses, and explanations for correctness and reproducibility.
• Flagging Python ML recommendations that are misleading, overconfident, statistically invalid, non-reproducible, insecure, or not production-ready.
• Communicating with teams, providing clear technical feedback, and updating contributors on guideline changes and review standards via Discord.
• Responding to questions regarding Python syntax, package usage, data leakage, model validation, metrics, statistical assumptions, reproducibility, and rubric interpretation.
• Managing inactive contributors, encouraging engagement, tracking follow-ups, and flagging availability issues.
• Creating and maintaining Python ML style guides, trackers, FAQs, examples, honeypots, calibration tasks, and onboarding materials.
• Identifying recurring quality gaps and developing scalable QA processes.
• Conducting onboarding and training sessions for Python ML contributors.

Qualifications and Experience

Candidates should possess the following qualifications:

• A Bachelor’s, Master’s, or PhD degree in Computer Science, Machine Learning, Data Science, Statistics, Mathematics, Engineering, or a closely related quantitative field.
• A minimum of 3 years of professional experience in Python development, machine learning, data science, ML engineering, model evaluation, research engineering, technical review, or ML education.
• Experience leading or supporting remote teams of trainers, annotators, reviewers, engineers, data scientists, ML researchers, coding mentors, or QAs is strongly preferred.
• Experience with AI training, data annotation, LLM evaluation, code QA, ML QA, or rubric-based technical review is a strong plus.

Technical and Soft Skills

Required skills include:

• A strong understanding of Python fundamentals including data structures, functions, classes, iterators, comprehensions, exception handling, virtual environments, package management, testing, and debugging.
• A strong understanding of ML topics such as supervised/unsupervised learning, feature engineering, train/test splits, cross-validation, model selection, data leakage, regression, classification, clustering, metrics, bias/variance, regularization, and reproducibility.
• The ability to evaluate ML content against detailed rubrics and identify issues such as flawed methodology, incorrect metrics, data leakage, non-reproducible code, invalid assumptions, hallucinated APIs, misleading conclusions, or incomplete explanations.
• Familiarity with libraries and tools such as NumPy, pandas, scikit-learn, PyTorch, TensorFlow/Keras, XGBoost/LightGBM, Jupyter, matplotlib, seaborn, MLflow, Hugging Face, SQL, GitHub, Docker, and CI/CD is preferred.
• Strong English language proficiency for following guidelines, communicating with teams, and providing clear technical feedback.
• Comfort working in fast-moving remote environments using platforms like Discord, Google Sheets, Google Docs, trackers, dashboards, GitHub, and project management systems.
• High organizational skills to maintain documentation and quality control materials.

Work Arrangement and Process

This is an hourly, remote contractor role. The selection process involves an AI interview, a domain-specific task, and an interview with a recruiter. While there is no immediate project for this role, qualified candidates will be prioritized for future relevant opportunities and gain access to the company's expert network.