Data Engineer (Spark ,Scala, Elastic Search)
Unison Consulting Pte Ltd, Taiwan

• Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
• Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
• Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
• Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
• Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
• Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
• Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration

Requirements

1. 5+ Years as Data Engineer
2. 5 Years experience in Spark, Scala, Elastic Search
3. Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
4. Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
5. Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation

Job Rewards and Benefits