Lead AI Data Engineer

About the Role

Standard Chartered is seeking a Lead AI Data Engineer with 5-10 years of experience to build and manage data pipelines and infrastructure for AI/ML and agentic applications. This permanent role involves owning the data lifecycle from ingestion to retrieval-ready and feature-ready states.

Key Responsibilities

• Build and operate production data pipelines at scale, managing schema changes, backfills, and on-call duties.
• Utilize Spark and Databricks (or equivalent distributed compute) to tune jobs for skew, shuffle, partitioning, and memory within a lakehouse platform (Delta, Unity Catalog, workflows).
• Manage object stores (S3 / ADLS / MinIO) as the data backbone, with a strong understanding of partitioning, file formats (Parquet / Delta / Iceberg), compaction, and performance tradeoffs.
• Integrate data across heterogeneous sources including databases, APIs, event streams, and files into reliable flows.
• Process unstructured data by parsing, chunking, and normalizing documents and multimodal content for AI systems.
• Design systems with clear reasoning on throughput, latency, consistency, idempotency, and cost.
• Participate in and drive deliverables for business use cases as a member of the AI team.
• Understand business requirements, execute solutioning, and ensure timely delivery commitments.
• Design and deliver AI/ML use cases, ensuring RAI, security, and governance.
• Manage model validation and improvements, and engage in stakeholder management.
• Manage project assignments and deadlines for the team.
• Oversee a team dedicated to reviewing models related to unstructured and structured data.
• Hire and nurture talent as required.
• Own delivery, highlight risks to stakeholders in a timely manner, and identify remediation plans with a risk roadmap.

Required Skills and Experience

• Experience: 5-10 years in data engineering, specifically with production data pipelines at scale.
• Distributed Compute: Proficiency with Spark and Databricks (or equivalent), including job tuning and working within lakehouse platforms.
• Data Storage: Strong grasp of object stores (S3 / ADLS / MinIO), partitioning, file formats (Parquet / Delta / Iceberg), compaction, and cost/performance tradeoffs.
• Data Integration: Ability to integrate data from heterogeneous sources (databases, APIs, event streams, files).
• Unstructured Data: Experience parsing, chunking, and normalizing documents and multimodal content for AI systems.
• System Design: Clear reasoning about throughput, latency, consistency, idempotence, and cost.

Preferred Qualifications

• Experience with feature engineering for ML, including building and serving features with offline/online consistency.
• Familiarity with streaming and big-data stacks such as Kafka / Kinesis / Pulsar, Flink, and lakehouse patterns.
• Contributions to open-source projects.
• Ability to coach and mentor team members in building scalable machine learning solutions.
• Strong communication skills and an easy-going attitude.
• Experience building and managing relationships with internal and external stakeholders.
• Familiarity with RAG pipelines, including embedding generation, vector stores, and retrieval quality.
• Knowledge of agentic integrations, tools, and data interfaces for autonomous AI workflows.

Governance and Conduct

The successful candidate will demonstrate awareness and understanding of the regulatory framework in which Standard Chartered operates, along with relevant regulatory requirements and expectations. This includes displaying exemplary conduct, adhering to the Group’s Values and Code of Conduct, and taking personal responsibility for embedding high standards of ethics. The role requires effective and collaborative identification, escalation, mitigation, and resolution of risk, conduct, and compliance matters.

Stakeholder Engagement

This role involves collaboration with various key stakeholders, including Business Stakeholders, the AIML Engineering Team, the AIML Product Team, the Product Enablement Team, the SCB Infrastructure Team, and the Interfacing Program Team.