Data Engineer (Temporary position)

Job Summary:

We are looking for a Mid-level Data Engineer to join a small, focused team building and maintaining a production-grade financial data processing platform for the Abu Dhabi Securities Exchange (ADX).

The platform ingests daily stock trading data from ADX, processes it through a series of ETL pipelines, tracks shareholder balances and positions, and powers downstream analytics via PostgreSQL and Elasticsearch. The codebase is Python-first, well-structured, and held to high quality standards (strict typing, clean architecture, comprehensive tests).

You will own the data and application layer — designing pipelines, evolving the schema, writing business logic services, and keeping the system accurate and reliable. The CTO handles all infrastructure and deployment concerns.

Job Responsibilities:

●    Build and maintain ETL pipelines that ingest daily Excel/CSV trade files, transform and validate the data, and persist it to PostgreSQL

●    Design and evolve database schemas using SQLAlchemy ORM and Alembic migrations

●    Implement business logic services: shareholder balance tracking, sharebook reconciliation, movement detection, classification rules

●    Maintain and extend Elasticsearch sync services (incremental and full resync workflows)

●    Support business stakeholders with data questions using the ELK stack — writing and running queries, interpreting results, and surfacing relevant data from Kibana

●    Parse and process structured financial data files (Excel, CSV) using Pandas, including data validation and deduplication

●    Write robust, type-safe Python following project standards (mypy strict mode, Black, isort)

●    Write and maintain automated tests with pytest — unit tests, repository mocks, snapshot tests

●    Collaborate with the CTO on data flow design and review migration strategies before deployment

Required Skills and Qualifications:

●    3–5 years of professional software/data engineering experience

●    At least 2 years working with Python in a production environment

●    Python 3.8+ — 3+ years of production experience; comfortable with type hints, async/await, and clean OOP design

●    PostgreSQL — schema design, indexing strategies, query optimization, migrations

●    SQLAlchemy 2.0 — declarative ORM models, session management, Alembic for migrations

●    Pandas — data transformation, Excel/CSV parsing, validation pipelines

●    FastAPI — building and maintaining REST APIs with dependency injection and async handlers

●    Pytest — writing unit and integration tests, using fixtures and mocking patterns

●    ELK Stack (Elasticsearch, Kibana) — working knowledge of Elasticsearch indexing and querying; able to write queries and navigate Kibana to answer business questions (dashboard building not required; Logstash not used)

●    Code quality discipline — strong typing, Black formatting, readable and maintainable code

Nice to Have

●    Web scraping — familiarity with Playwright or similar browser automation tools

●    Financial data domain — understanding of stock trading concepts (OHLCV data, shareholder registers, portfolio performance metrics such as XIRR or TWR)

●    Financial APIs — experience with yfinance or similar market data sources