Senior Data Engineer (Databricks and Pyspark)

Position : Senior Data Engineer.
Location : Malaysia and Singapore.
Language : Mandarin Language must.
Role : Full Time Perm role + one month Bonus.

Requirements

We are looking for aSenior Data Engineerwith strong expertise inDatabricks, PySpark, Delta Lake, and cloud-based data pipelines. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.

Key Responsibilities

• Design, build, and optimizedata ingestion and transformation pipelinesusing Databricks, PySpark, and Python.
• ImplementDelta Lake and Medallion architecturefor scalable enterprise data platforms.
• Develop ingestion frameworks for data fromSFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
• Automate workflows usingDatabricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
• Optimize Spark jobs for performance, reliability, and cost efficiency.
• Implementdata validation, quality checks, and monitoringwith automated alerts and retries.
• Design secure and governed datasets usingUnity Catalogand cloud security best practices.
• Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
• Provide technical leadership and guidance to junior team members.

Required Skills

• 5–8+ years of experience inData Engineering.
• Strong hands-on experience withDatabricks, PySpark, Delta Lake, SQL, Python.
• Experience withAzure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
• Experience integrating data fromAPIs, SFTP servers, vendor data providers, and cloud storage.
• Knowledge ofETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
• Strong experience withGit, Azure DevOps CI/CD, and YAML pipelines.
• Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).

Good to Have

• Exposure toOil & Gas or trading analytics(SPARTA, KPLER, IIR, OPEC).
• Knowledge ofPower BIor data visualization concepts.
• Familiarity withTerraform, Scala, or PostgreSQL.
• Experience with SharePoint development or .NET (optional)