BTL / ETL - Data Engineer - SQL / DBT In Azure Databricks

Responsibilities

1. Design, Develop and maintain Business Translation Layer (BTL) pipelines using SQL/DBT in Azure Databricks 


2. Create build from Github and release pipeline and Databricks using Azure Devops / Harness


3. Monitor Performance of BTL Jobs, resolve any issues and improve the performance metrics as needed.


4. Diagnose system performance issue related to data processing and implement solution to address 


5. Collaborate with other teams to ensure successful integration of data pipelines into larger system architecture requirement.


6. Maintain integrity and quality across all pipelines and environments.


7. Understand and follow secure coding practice to make sure code is not vulnerable 

Responsibilities

1. Design, Develop and maintain Business Translation Layer (BTL) pipelines using SQL/DBT in Azure Databricks 


2. Create build from Github and release pipeline and Databricks using Azure Devops / Harness


3. Monitor Performance of BTL Jobs, resolve any issues and improve the performance metrics as needed.


4. Diagnose system performance issue related to data processing and implement solution to address 


5. Collaborate with other teams to ensure successful integration of data pipelines into larger system architecture requirement.


6. Maintain integrity and quality across all pipelines and environments.


7. Understand and follow secure coding practice to make sure code is not vulnerable 

Skills

• Should have Strong background in Software development with experience in ingest, transform and store data from large datasets using SQL with strong knowledge on distributed computing concepts.


• Must have hands on experience in designing and developing Data Transformation Logic using SQL


• Must have hands on Experience with Databricks.


• Must have minimum 5 years of exposure and good proficiency in data warehousing concepts.


• Extensive experience writing, optimizing, and data pipelines.


• Proficient in SQL and database Design concepts.


• Knowledge of Azure cloud computing platform with Azure Synapse, ADLS


• Knowledge on GitHub and build management.


• Passion for data and experience working within a data driven organization


• Excellent presentation, communication (oral & written), and relationship building skills, across all levels of management