Responsibilities:
• Build and optimize ETL/ELT pipelines for large volumes of data.
• Implement Delta Lake structures and data models for operational and analytical consumption.
• Develop incremental loading, historization, and CDC (Change Data Capture) mechanisms.
• Integrate data from multiple sources, including Oracle, SQL Server, APIs, files, banking applications, etc.
• Participate in defining the Data Lakehouse architecture.
• Propose scalable and high-performance data models.
• Contribute to the development of development standards and platform architecture.
• Ensure data traceability and consistency throughout the entire processing flow.
• Implement automated Data Quality controls.
• Develop data reconciliation and validation processes.
• Configure monitoring, alerting, and process logging.
• Analyze and resolve production incidents.
• Manage data reloads, reruns, corrections, and recovery processes.
• Optimize PySpark and SQL processes for cost efficiency and performance.
• Identify and eliminate processing bottlenecks.
• Optimize Databricks resource consumption.
• Contribute to the automation of development and deployment processes.
Qualifications:
• Minimum 2–3 years of hands-on experience with Databricks.
• Proven experience working on complex Data Warehouse, Data Mart, or Data Lake projects.
• Experience in developing end-to-end data pipelines.
• Strong knowledge of Data Lakehouse architecture.
• Experience implementing historization, CDC, data reconciliation, and Data Quality processes.
• Ability to take full ownership of technical deliveries and provide technical leadership to the team.
• Technical skills: Azure Databricks / PySpark, Advanced SQL, Delta Lake, Git & Jira, Data Modelling, ETL / ELT,
Data Quality Frameworks, Pipeline Orchestration.
*We are an equal opportunity employer and value diversity. All employment decisions are made without regard to age, gender, disability, race, ethnicity, religion, sexual orientation, or any other protected characteristic. We encourage applicants from all backgrounds to apply.*