Data Engineer

Role and Responsibilities

Data Engineer (AWS, Databricks, Python and Informatica )

A Cloud Data Engineer specializing in AWS, Databricks, and Informatica IDMC is responsible for building and maintaining a robust, integrated, and governed data infrastructure that leverages the strengths of the analytics platforms to extract valuable insights from data while ensuring data security, compliance, and high-quality data management.

Roles And Responsibilities:

1. Design engineering solutions, including databases, data lakes, and warehouses, using AWS services along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging.
2. Create, manage, and optimize data pipelines for ingesting, processing, and transforming data using AWS services like AWS Glue, AWS Data Pipeline, and AWS Lambda, Databricks for advanced data processing, and Informatica IDMC for data integration and quality.
3. Integrate data from various sources, both internal and external, into AWS and Databricks environments, ensuring data consistency and quality, while leveraging Informatica IDMC for data integration, transformation, and governance.

Requirements / Qualifications

1. Bachelor’s or master’s degree in computer science, data engineering, or a related field.
2. Minimum 4 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC.
3. Proficiency in programming languages such as Python, Java, or Scala for building data pipelines.
4. Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
5. Strong knowledge of SQL and NoSQL databases.
6. Familiarity with data modeling and schema design.
7. Excellent problem-solving and analytical skills.
8. Strong communication and collaboration skills.