ETL QA

Role Overview

We are looking for an ETL QA Engineer with strong experience in
Databricks and PySpark, combined with hands-on exposure to AI-assisted
development and prompt engineering. The ideal candidate will be responsible for
validating large-scale data pipelines, ensuring data quality, and leveraging
tools like GitHub Copilot, GitHub Actions/Runners, and AI agents to improve
testing efficiency and coverage. You will also contribute to building and
validating AI-powered knowledge bases.

Required Skills & Experience

• Core
  Technical Skills
  
• Strong hands-on experience in Databricks (notebooks, jobs, clusters, Delta Lake, SQL).
  
• Expert-level PySpark skills, including transformations, actions, window functions, and
  optimization techniques.
  
• Solid understanding of ETL/ELT
  concepts, data warehousing, and data modeling (star/snowflake
  schemas, dimension/fact tables).
  
• Proven experience in data
  validation, data quality checks, and reconciliation techniques.
  
• Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CD
  pipelines.
  
• AI &
  Prompt Engineering
  
• Experience crafting effective
  prompts for large language models (LLMs) for coding, testing, and
  documentation.
  
• Exposure to agent-based
  architectures or tools (e.g., workflow/agent frameworks that
  orchestrate multi-step AI tasks).
  
• Familiarity with AI-based
  tools for code analysis, test generation, or knowledge management.
  
• Knowledge
  Base & Documentation
  
• Experience building AI-driven
  knowledge bases or documentation systems (e.g., using vector search,
  embeddings, or LLM-based retrieval).
  
• Strong documentation skills,
  with the ability to translate complex data/QA concepts into clear
  knowledge articles.
  
• Testing & QA Practices
  
• Strong understanding of QA
  methodologies: test planning, test design, defect management, and
  regression testing.
  
• Experience with automated
  testing frameworks (Python-based testing libraries such as pytest, unittest, or similar).
  
• Familiarity with data quality
  tools/practices (e.g., validation rules, thresholds, anomaly detection).
  

Key Attributes

• Strong analytical and
  problem-solving skills with a detail-oriented mindset.
  
• Passion for data quality,
  automation, and continuous improvement.
  
• Ability to work in an agile,
  fast-paced environment and collaborate across multiple teams like Business
  and operations team and AD team.
  
• Curiosity and openness to adopting new AI tools and
  practices for QA.