Data Engineer

Job Description

**Spark Scala Engineer – 12 month initial contract – Leeds (3 days in the office) - £550 per day (Inside IR35)**

We are looking for a Spark Scala Engineer to join a long-term project for a management consulting firm working with one of the biggest financial services organisations in the world.
The successful candidate will be responsible for designing, building, and maintaining data pipelines using Apache Spark and Scala. This includes tasks like:
Extracting data from various sources (databases, APIs, files)
Transforming and cleaning the data
Loading the data into data warehouses or data lakes (e.g., BigQuery, Amazon Redshift)
Automating the data pipeline execution using scheduling tools (e.g., Airflow)
Work with Big Data technologies: You'll likely work with various Big Data technologies alongside Spark, including:
Hadoop Distributed File System (HDFS) for storing large datasets
Apache Kafka for real-time data streaming
Apache Hive for data warehousing on top of HDFS
Cloud platforms like AWS, Azure, or GCP for deploying and managing your data pipelines

Key skills:
At least 8+ Years of IT Experience with designing, building, and maintaining data pipelines.
At least 4+ Years of experience with designing, building, and maintaining data pipelines using Apache Spark and Scala
Programming languages: Proficiency in Scala and Spark is essential. Familiarity with Python and SQL is often a plus.
Big Data technologies: Understanding of HDFS, Kafka, Hive, and cloud platforms is valuable.
Data engineering concepts: Knowledge of data warehousing, data pipelines, data modelling, and data cleansing techniques is crucial.
Problem-solving and analytical skills: You should be able to analyse complex data problems, design efficient solutions, and troubleshoot issues.
Communication and collaboration: The ability to communicate effectively with data scientists, analysts, and business stakeholders is essential.

Beneficial skills:
Machine learning libraries: Familiarity with Spark ML or other machine learning libraries in Scala can be advantageous.
Cloud computing experience: Experience with cloud platforms like AWS, Azure, or GCP for data pipelines deployment is a plus.
DevOps tools: Knowledge of DevOps tools like Git, CI/CD pipelines, and containerization tools (Docker, Kubernetes) can be beneficial.