Quantexa Data Engineer

Role Overview

We are seeking a talented and experiencedData Engineerwith strong expertise inQuantexa, Hadoop, Scala, Apache Spark, Elasticsearch, OpenShift Container Platform (OCP), and DevOps practices.

The successful candidate will be responsible for designing, developing, optimizing, and maintaining scalable big data solutions usingApache Spark, Scala, Hadoop, and Elasticsearch. The role involves collaborating with cross-functional teams to build efficient data processing pipelines and search applications.

Knowledge and experience in the Compliance / AML domain will be an added advantage.

Key Responsibilities

• Design, develop, and implementSpark/Scala applicationsand data processing pipelines for large volumes of structured and unstructured data.
• Implementdata transformation, aggregation, enrichment, and computationprocesses to support analytics and machine learning initiatives.
• Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions.
• IntegrateElasticsearch with Sparkfor efficient data indexing, querying, and retrieval.
• Implement transformations and aggregations usingSpark RDDs, DataFrames, Datasets, and Spark SQL.
• Develop scalable, reliable, and fault-tolerant Spark applications following industry best practices and coding standards.
• Optimize and tuneSpark jobs and Elasticsearch queriesto improve performance, scalability, and resource utilization.
• Monitor job performance, identify bottlenecks, troubleshoot issues, and implement appropriate optimizations.
• Troubleshoot and resolve issues related todata processing, data quality, Spark performance, and Elasticsearch integration.
• Ensure data quality, consistency, accuracy, and integrity throughout the data processing lifecycle.
• Design and deploy data engineering solutions onOpenShift Container Platform (OCP)using containerization and orchestration technologies.
• Optimize data engineering workflows for containerized environments and efficient resource utilization.
• Collaborate with DevOps teams to streamline deployments and implementCI/CD pipelines.
• Implementdata governance, data lineage, and metadata managementpractices to ensure data accuracy, traceability, and compliance.
• Implement monitoring and logging mechanisms to ensure the health, availability, and performance of data infrastructure.
• Monitor and optimize end-to-end data pipeline performance and implement required enhancements.
• Document data engineering processes, workflows, architecture, and infrastructure configurations for knowledge sharing and future reference.

Requirements

• Bachelor’s or Master’s degree inComputer Science, Software Engineering, Information Technology, or a related field.
• Quantexa Certified Data Engineer / Data Architectwith hands-on experience and strong proficiency in the Quantexa platform.
• Proven experience as aData Engineerworking with Hadoop, Spark, and large-scale data processing technologies.
• Strong proficiency inScalaand familiarity with functional programming concepts.
• In-depth understanding ofApache Spark architecture, RDDs, DataFrames, Datasets, and Spark SQL.
• Strong expertise inHadoop ecosystem technologies, including HDFS, Hive, Pig, and related tools.
• Hands-on experience withElasticsearch, including data indexing, search applications, data modeling, indexing strategies, and query optimization.
• Experience withOpenShift Container Platform (OCP)and Kubernetes-based container orchestration.
• Strong programming skills inScala, Python, Java, and/or Spark.
• Good understanding ofDevOps practices, CI/CD pipelines, and infrastructure automation.
• Experience with tools such asDocker, Jenkins, Ansible, and Bitbucket.
• Experience withdistributed computing, parallel processing, and large-scale datasets.
• Strong experience inperformance tuning and optimizationof Spark applications and Elasticsearch queries.
• Experience withGitand collaborative software development workflows.
• Strong analytical and problem-solving skills with the ability to troubleshoot complex technical issues.
• Excellent communication and collaboration skills with the ability to work effectively with cross-functional teams.
• Experience withGrafana, Prometheus, and Splunkwill be an added advantage.
• Exposure to cloud platforms such asAWS, Azure, or GCPand their data services will be a plus.
• Knowledge or experience in theCompliance / AML domainwill be an added advantage.