Senior Data Engineer - Databricks

Accountabilities

• Design, develop, maintain, and optimize scalable data pipelines using Databricks, PySpark, Delta Lake, and SQL.

• Integrate data from diverse sources, including APIs, relational databases, files, and other enterprise systems.

• Build and maintain robust data ingestion and transformation processes aligned with the Medallion architecture and modern data engineering practices.

• Collaborate with business and analytics teams to understand data requirements and deliver solutions that support strategic objectives.

• Ensure the quality, reliability, security, and governance of ingested and processed data throughout the data lifecycle.

• Apply modern DataOps practices, including code versioning, automated data testing, CI/CD pipelines, and infrastructure as code.

• Troubleshoot and resolve performance, scalability, and reliability issues across data pipelines and distributed processing environments.

• Document data engineering processes, technical decisions, standards, and best practices to support maintainability and knowledge sharing.

• Provide technical guidance and mentorship to colleagues while collaborating with data scientists, analysts, and platform engineers.

• Contribute to continuous improvement in data architecture, pipeline efficiency, and engineering quality.

Requirements:

• Proven experience designing, developing, and managing large-scale data pipelines using Databricks, PySpark, Delta Lake, and SQL.

• Strong programming skills in Python and SQL for data processing, transformation, and pipeline development.

• In-depth understanding of ETL/ELT frameworks, data warehousing concepts, and distributed data processing.

• Hands-on experience with modern DataOps practices, including Git version control, CI/CD pipelines, automated testing, and infrastructure as code.

• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform and their associated data services.

• Strong problem-solving and analytical skills, with the ability to troubleshoot and optimize data solutions for performance, scalability, and reliability.

• Proficiency with Git and collaborative software development workflows.

• Advanced English proficiency for effective communication, technical collaboration, and documentation.

• Nice to have:Experience with data contracts, schema evolution, and maintaining compatibility between services and data consumers.

• Familiarity with data quality frameworks such as Great Expectations, Soda, dbt tests, or custom-built validation solutions.

• Experience integrating data solutions with Power BI.

Benefits:

• Health and dental insurance.

• Meal and food allowances.

• Childcare assistance.

• Extended paternity leave.

• Access to gyms and health and wellness professionals through Wellhub (Gympass) and TotalPass partnerships.

• Profit Sharing and Results Participation (PLR).

• Life insurance.

• Access to a continuous learning platform and professional development resources.

• Discount club with partner offers.

• Free online resources dedicated to physical, mental, and overall well-being.

• Pregnancy and responsible parenting courses.

• Partnerships with online learning platforms.

• Access to a language-learning platform.

• Additional benefits and well-being initiatives.