Senior Data Engineer

Role Overview

DigyCorp is seeking a Senior Data Engineer for a contract position in Makkah, specifically in Thuwal or Makkah city. This role involves leading data engineering efforts for a petabyte-scale industrial digital twin, central to a national initiative focused on coral restoration. The position requires working with imagery, geospatial data, and sensor telemetry, and involves building curated datasets for production AI and MLOps pipelines, as well as establishing data governance frameworks.

Key Responsibilities

• Build and maintain bronze/silver/gold pipelines in Databricks, processing various source formats into curated, analysis-ready datasets.
• Develop Databricks notebooks/jobs (PySpark/SQL), manage job scheduling, and tune performance.
• Register curated datasets in Unity Catalog with appropriate metadata, ownership, and access tags.
• Apply data modeling and data warehousing best practices when designing silver/gold layer schemas.
• Utilize Databricks Lakebase for transactional/OLTP-style access alongside Delta Lake data.
• Build and maintain Azure Data Factory pipelines and triggers for data movement from source systems to Databricks.
• Consume streaming data from Azure Event Hubs and configure triggers for data ingestion.
• Develop Azure Function Apps for processing incoming Event Hub data and initiating downstream processes via Batch Accounts.
• Manage Azure Batch Accounts for data reception from third-party tools, IoT/sensor feeds, and external sources, moving data into Databricks.
• Ensure the ingestion chain (Event Hub → Function App → Batch Account → Databricks) is resilient, monitored, and recoverable.
• Migrate data from various sources/platforms into Databricks, preserving integrity and minimizing disruption.
• Support migration of large imagery/scientific-archive data, including metadata preservation and integrity validation.
• Manage credentials and secrets in Azure Key Vault and implement access controls (RBAC, managed identities).
• Maintain Azure Data Lake as the platform's storage layer, including file and metadata organization.

Downstream Enablement and Data Architecture

• Prepare curated, governed datasets for Power BI/Tableau reporting and Digital Twin applications.
• Prepare feature-ready datasets for AI/ML consumption and support MLflow-tracked training and inference data needs.
• Build and maintain the feature store and feature-engineering pipelines, ensuring models consume governed, versioned datasets.
• Provide the data foundation for MLOps, including reproducible datasets, versioning, lineage, and drift/freshness monitoring.
• Build data pipelines for GenAI and RAG use cases, including document preparation, embedding generation, vector index management, and retrieval source refreshing.
• Enable advanced analytics capabilities, including graph and knowledge-graph data structures for simulation, prediction, and decision support.
• Work with geospatial (PostGIS) and time-series/sensor data feeds within the ingestion and curation pipeline.
• Own the Lakehouse data architecture, including medallion layering, domain boundaries, curated data products, and serving patterns.
• Own application-level data architecture across all current and future applications.
• Advise on target-state decisions, ensuring ingestion, curation, reporting, and ML consumption are architected as a unified estate.

Data Governance Leadership

• Run the initial data assessment and define the governance framework, including the data domain map, ownership register, and standards for data quality, classification-driven access, privacy, metadata, and lifecycle.
• Implement the governance framework within the platform, including data quality rules with live issue routing, QA/QC split with the business, and retention/deletion processes.
• Own the implementation of Data Quality rules on relevant applications and direct development teams on necessary controls.
• Integrate newly onboarded domains into the framework and metadata standard, demonstrating operational governance.

Qualifications and Experience

• 5-10 years of experience in data engineering.
• Demonstrated experience with Databricks Lakehouse architecture (bronze/silver/gold layers).
• Proficiency in Azure Data Factory, Event Hubs, Function Apps, and Batch Accounts.
• Experience with data migration, security, and storage in Azure environments.
• Strong understanding of data governance principles and implementation.
• Ability to work in a hands-on delivery-first role with advisory responsibilities.

Work Type and Location

This is a contract position based in Makkah, Saudi Arabia, with options to work in Thuwal or Makkah city. The role involves working within a small, senior team where decisions are quickly implemented into production.