Data Engineer (Quantexa)

<ul>
<li>Implement data transformation, aggregation, and enrichment processes to support various data analytics and machine learning initiatives</li>
<li>Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions</li>
<li>Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data</li>
<li>Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data</li>
<li>Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch</li>
<li>Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch</li>
<li>Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards</li>
<li>Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration</li>
<li>Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components</li>
<li>Ensure data quality and integrity throughout the data processing lifecycle</li>
<li>Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques</li>
</ul><h2>Requirements</h2><ul>
<li>Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field</li>
<li>Must be Quantexa certified data engineer / data architect and proficient with the software.</li>
<li>Proven experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments</li>
<li>Proficiency in Scala programming language and familiarity with functional programming concepts</li>
<li>Experience with Quantexa tool is highly preferred.</li>
<li>In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL</li>
<li>Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Pig, etc)</li>
<li>Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes</li>
<li>Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java</li>
<li>Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)</li>
<li>Experience with Graphana, Prometheus, Splunk will be an added benefit</li>
<li>Experience integrating and working with Elasticsearch for data indexing and search applications</li>
<li>Experience with Quantexa tool is highly preferred.</li>
<li>In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL</li>
<li>Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Pig, etc)</li>
</ul>