Data Engineer

Data Engineer

As a Principal Data Engineer, your responsibilities will include:

• Design and build data pipelines to process terabytes of data
  


• Orchestrate in Airflow the data tasks to run on Kubernetes/Hadoop for the ingestion, processing and cleaning of data.
  


• Create Docker images for various applications and deploy them on Kubernetes
  


• Design and build best in class processes to clean and standardize data.
  


• Troubleshoot production issues in our Elastic Environment
  


• Tuning and optimizing data processes
  

·        Advancing the team’s DataOps culture (CI/CD, Orchestration, Testing, Monitoring) and building out standard development patterns

• Drive innovation by testing new technology and approaches to continually advance the capability of the data engineering function.
  


• Drive efficiencies in current engineering processes via standardization and migration of existing on-premise processes to the cloud
  


• Ensuring Data Quality – building best in class data quality monitoring that ensure that all data products exceed customer expectations.
  

Required Qualifications:

• Computer Science bachelor’s degree or similar.
  


• Good understanding of Data Modelling techniques i.e. DataVault, Kimble Star
  


• Excellent understanding of Column-Store RDBMS (DataBricks, Snowflake, Redshift, Vertica, Clickhouse)
  


• Good experience handling real-time, near real-time and batch data ingestions
  


• Hands on experience on the following technologies:
  




• Developing processes in Spark
  


• Writing complex SQL queries f
  


• Building ETL/data pipelines
  


• Exposure to Kubernetes and Linux containers (i.e. Docker)
  


• Related/complementary open source software platforms and languages (e.g. Scala, Python, Java, Linux)
  




• Proven track record of designing effective data strategies and leveraging modern data architectures that resulted in business value 
  


• Experience building cloud-native data pipelines on either AWS, Azure or GCP, following best practices in cloud deployments
  

·        Strong DataOps experience (CI/CD, Orchestration, Testing, Monitoring)

• Strong experience leading and developing data engineering teams
  

·        Demonstrated effective interpersonal, influence, collaboration and listening skills

·        Strong stakeholder management skills

·        Excellent time management, organizational and prioritization skills with ability to balance multiple priorities.

Preferred Qualifications:

·        Experience with data tokenization and different techniques and tools i.e. DataVant, Protegrity

·        Experience with Azure Data Factory, Databricks and Snowflake

• Experience with Apache Spark and related Big Data stack and technologies, PySpark Scala
  

·        Experience working with Apache Kafka, building appropriate producer/consumer apps

·        Experience working with Kubernetes and Docker, and knowledgeable about cloud infrastructure automation and management (e.g., Terraform)

·        Experience working in projects with agile/scrum methodologies

·        Familiarity with production quality ML and/or AI model development and deployment.

·        Healthcare industry knowledge and experience with exposure to EDI, HIPAA, HL7 and FHIR integration standards