Data Engineer

As aData Engineer, your main goal is tobuild and maintain the systems that process and store datafor the Events & Exhibitions ecosystem. You willtake scattered, vendor-specific data(from registration systems, apps, marketing tools) andtransform it into a unified, AI-ready datasetusing aMedallion Architectureon Azure.

Think of it asorganizing raw data into a structured pipelinethat’s ready for analysis and machine learning.

Requirements

1. Data Ingestion & API Integration (Bronze Layer)

• Build and manage robust ETL/ELT pipelines usingAzure Data Factoryto ingest data from 3rd-party vendors (REST APIs, Webhooks, SFTP).
• Ensure raw data is landed securely inAzure Data Lake Gen2(Bronze Layer) without data loss.
• Implement error-handling and logging to monitor the health of real-time and batch ingestion jobs.

2. Transformation & Modeling (Silver & Gold Layers)

• UtilizePySpark (Azure Databricks/Synapse)andSQLto clean, deduplicate, and standardize data in the Silver Layer.
• ExecuteIdentity Resolutionlogic to stitch together visitor and exhibitor profiles from multiple touchpoints into a "Golden Record."
• Develop optimized data sets in the Gold Layer for high-performance reporting and predictive AI models.

3. Infrastructure & Performance Optimization

• Optimize SQL queries and Spark jobs to reduce Azure compute costs and minimize data latency.
• Maintain theData Dictionaryand technical documentation to ensure the "Engine Room" logic is transparent and scalable.
• Implement data masking and security protocols to ensure GDPR and internal compliance.

4. Business Enablement

• Support the Senior Data Manager in building theSemantic Layerthat feeds our Power BI "Data Window."
• Collaborate with the Events Tech team to troubleshoot data discrepancies between front-end apps and back-end tables.

Technical Requirements

• Experience:3–5 years in Data Engineering with a focus on Cloud environments.
• Core Azure Stack:Proven expertise inAzure Data Factory, Azure Synapse Analytics, and Data Lake Gen2.
• Coding:High proficiency inSQL(complex joins/optimizations) andPython/PySpark.
• Architectural Knowledge:Practical experience with theMedallion Architecture(Bronze/Silver/Gold).
• Integration:Strong experience working with REST APIs and JSON/XML data formats.

Benefits