Senior Data Engineer

Summary

Responsibilities

• Design and implement scalable data models and comprehensive data architectures to support enterprise needs.
  
• Establish and enforce rigorous data validation protocols to ensure high data quality and accuracy across all systems.
  
• Develop sophisticated error handling mechanisms and logging frameworks to monitor pipeline health and troubleshoot issues efficiently.
  
• Orchestrate complex data workflows and scheduling using tools such as Apache Airflow, Tidal, or Control-M.
  
• Manage version control and automate deployment processes through Git and continuous integration/continuous deployment (CI/CD) pipelines.
  
• Build and optimize Extract, Transform, Load (ETL) processes to facilitate seamless data movement and transformation.
  
• Utilize Databricks, PySpark, and Spark SQL to process large-scale datasets and perform advanced data engineering tasks.
  

• Possess 5 to 7 years of professional experience in data engineering roles within the IT Services sector.
  
• Demonstrate strong proficiency in Data Modeling and Data Architecture principles.
  
• Show proven expertise in Data Validation and Data Quality assurance methodologies.
  
• Have hands-on experience with Error Handling and Logging strategies in production environments.
  
• Be skilled in orchestrating workflows using Apache Airflow, Tidal, or Control-M.
  
• Display competency in using Git for version control and managing CI/CD pipelines.
  
• Maintain a solid background in Data Engineering and ETL development.
  
• Possess practical experience with Databricks, PySpark, and Spark SQL technologies.