Senior Data Engineer

DVT, founded in 1999, helps businesses become and remain leaders in the market with high quality software solutions that add business value quicker and are released to production faster. Its services extend from custom software development to business software solutions consulting, software quality assurance, outsourced automated regression testing, enterprise mobility solutions, business intelligence solutions as well as Agile training and consulting. DVT has grown to over 600 staff with offices in the UK (London) and South Africa (Johannesburg, Centurion, Cape Town and Durban). The company services more than 100 local and international, medium and large organisationsJob description DVT is a global custom software development and data engineering company. With our remote and hybrid options, our vision is to be South Africa's favourite custom software solutions & services company, with a global footprint. You will have the opportunity to work alongside some of the most established developers in the country with the latest technologies. DVT is committed to continuously training our staff and we are very proud of our culture of learning, from internal speaking and training to sponsoring a variety of technical events from DevConf to GDG. The Senior Data Engineer will play a crucial role in designing, developing, and maintaining scalable data solutions and infrastructure for our clients. This position requires a deep understanding of data engineering principles, technologies, and tools, as well as strong problem-solving skills. The Senior Data Engineer will collaborate closely with cross-functional teams to ensure data pipelines are efficient, reliable, and aligned with business objectives within our clients. Job requirements

TECHNICAL KNOWLEDGE

Strong knowledge and extensive experience in: Proficient in big data technologies such as Apache Hadoop, Spark, and Kafka. Strong programming skills, particularly Python, Java, and SQL. Solid experience in developing and optimizing data pipelines, architecture, and data sets. Solid understanding of data warehousing concepts, ETL tools, and data modelling. Good experience with cloud-based data solutions like AWS (EC2, S3, EMR, Redshift, Glue), Google Cloud (BigQuery, Dataflow, Pub/Sub). Data pipeline orchestration and workflow management tools: Airflow, Luigi. Familiarity with Containerization and orchestration platforms: Docker, Kubernete

RESPONSIBILITIES

Works independently on complex data engineering projects. Develop and maintain optimal data pipelines architecture. Collaborate with data scientists and Business Analysts to meet functional data requirements. Assist the Lead Data Engineer in planning and implementing data engineering projects and be involved in project management and strategic decision-making. Optimize and tune data pipelines for performance, scalability, and cost-effectiveness. Ensure data quality and implement data validation and cleansing processes. Monitor and troubleshoot data pipelines to identify and resolve issues promptly. Evaluate and recommend appropriate data storage and processing technologies. Develop and maintain documentation for data engineering processes, data flows, and system architectures. Stay updated with emerging technologies, industry trends, and best practices in data engineering and analytics. Provide technical guidance and mentorship to junior data engineers.

MINIMUM EXPERIENCE REQUIRED

5+ years of experience in data engineering, with a focus on building large-scale data processing systems. Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.

BEHAVIOURAL COMPETENCIES

Shows promising leadership abilities. Excellent analytical and problem-solving skills Ability to work in a fast-paced, dynamic, and collaborative environment. Comfortable with ambiguity and able to adapt to change. Ability to work independently and as part of a team. Passion for learning new technologies and staying up to date with the latest trends in Data Engineering. Excellent written and verbal communication and collaboration skills Strong business acumen.