Job Description - Data Engineer (Spark ,Scala, Elastic Search)
Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
Requirements
5+ Years as Data Engineer
5 Years experience in Spark, Scala, Elastic Search
Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation
All Job Ads are subject to GrabJobs’s Terms of Service. We allow users to flag postings that may be in violation of those terms. Job Ads may also be flagged by GrabJobs moderation team. However, no moderation system is perfect, and flagging a posting does not ensure that it will be removed.
Be the first to receive the latest Others Full-Time Jobs in Australia.
Setup your job alert:
By activating job alerts, I agree to GrabJobs Terms & Privacy Policy. I can unsubscribe to job alerts anytime.
Skip
GrabJobs is the no1 job portal in Australia, connecting you to thousands of jobs fast!
Find the best jobs in Australia, apply in 1 click and get a job today!