Senior Data Engineer (Big Data, Data Lake & GenAI)

Job Overview

We are seeking an experienced Senior Data Engineer with strong expertise in enterprise Data Warehousing, Big Data platforms, Data Lake architectures, and modern cloud technologies. The ideal candidate will have hands-on experience designing and implementing large-scale ETL/ELT pipelines, data transformation frameworks, and analytics solutions within banking, financial services, or large enterprise environments.

The role requires deep technical knowledge of Apache Spark, PySpark, Hadoop ecosystems, Teradata, Cloud Data Platforms, and emerging Generative AI technologies for intelligent data processing and automation. The candidate should be capable of driving end-to-end data engineering initiatives, supporting business intelligence, analytics, regulatory reporting, and AI-enabled data solutions.

Key Responsibilities

• Design, develop, and optimize enterprise Data Warehouse and Data Lake solutions.
• Build scalable ETL/ELT pipelines using Python, Spark, and PySpark.
• Develop and maintain data ingestion, transformation, and analytical data processing frameworks.
• Implement data integration solutions across structured, semi-structured, and unstructured data sources.
• Manage large-scale data platforms using Hadoop, AWS S3, Azure Databricks, and Cloud Data Services.
• Develop data marts and business intelligence datasets for analytics and reporting.
• Implement metadata management, data lineage, and governance frameworks using tools such as Collibra and Enterprise Data Catalog.
• Design and optimize Spark SQL workloads for high-performance analytics.
• Build automated workflow orchestration solutions using Airflow, Control-M, or equivalent scheduling platforms.
• Integrate Generative AI capabilities into data engineering workflows for metadata tagging, data classification, anomaly detection, and intelligent reporting.
• Collaborate with business stakeholders, architects, data scientists, and analytics teams to deliver scalable data solutions.
• Ensure data quality, governance, security, and compliance requirements are met.
• Support production systems, troubleshoot performance issues, and maintain SLA compliance.

Required Skills

Data Engineering

• Apache Spark
• PySpark
• Spark SQL
• Python
• SQL / PL-SQL
• Hadoop HDFS
• Data Warehouse Design
• Data Lake Architecture
• ETL / ELT Development
• Dimensional Modeling
• Star Schema & Snowflake Schema

Cloud & Big Data

• AWS S3
• Azure Databricks
• GCP BigQuery
• Cloudera Data Platform
• Kafka
• Parquet
• Alluxio

Data Governance

• Collibra
• Enterprise Data Catalog (EDC)
• Metadata Management
• Data Lineage
• Data Quality Management

DevOps & Automation

• Airflow
• Control-M
• Jenkins
• GitLab
• Bitbucket
• CI/CD Pipelines

Generative AI

• OpenAI API
• Claude API
• LangChain
• Prompt Engineering
• Retrieval-Augmented Generation (RAG)
• Vector Databases (Pinecone, ChromaDB)

Preferred Experience

• 10+ years of Data Engineering experience.
• Banking, Financial Services, or Insurance domain experience.
• Experience working with Teradata Data Warehouses.
• Hands-on experience in Data Lake modernization initiatives.
• Experience delivering regulatory reporting and compliance data solutions.
• Exposure to AI-enabled data platforms and intelligent analytics solutions.
• Experience supporting enterprise-scale production environment