Lead Big Data Engineer with AI/ML experience

N-iX is looking for a Lead Big Data Engineer with AI/ML experience. As a Lead Engineer, you’ll be responsible for designing, developing, and optimizing data and AI pipelines (including Spark-based and RAG-style workflows), ensuring performance, scalability, and reliability. You’ll also be working on technical discovery and solution assessment: helping the client understand existing AI tooling (currently a RAG-based solution used for document analysis), evaluating scalability, quality, cost, and risks, and advising on future architecture options. 

Responsibilities:

• Design and implement data pipelines to support AI and ML use cases, including data preparation, feature engineering, and real-time model serving.


• Support and enable RAG-based and NLP use cases, including document ingestion, data preparation, feature extraction, and structured data generation.


• Collaborate with AI engineers to productionize AI/ML solutions and integrate them into reliable data workflows.


• Participate in and lead discovery phases to analyze existing AI solutions, clarify how they work, and identify architectural gaps, risks, and improvement opportunities.


• Assess scalability and multilingual support of AI/data solutions (including feasibility, configuration needs, cost, and expected quality).


• Contribute to evaluations of vendor lock-in, platform alternatives, and potential migration paths (e.g., toward Palantir Foundry or other modern data/AI platforms).


• Ensure high standards of data quality, security, governance, and compliance.


• Drive continuous improvement in development processes, tooling, and engineering practices.


• Foster collaboration across engineering, data science, and product/business stakeholders.

Requirements:

• 6+ years of experience in Data Engineering


• Strong hands-on experience with PySpark and large-scale data processing.


• Proficiency in Python and distributed computing concepts.


• Experience designing and optimizing scalable data pipelines for high-volume data.


• Experience supporting AI/ML projects (e.g., enabling model training pipelines, feature engineering, real-time inference, or MLOps workflows).


• Ability to perform technical discovery and solution assessment, including explaining complex systems to non-technical stakeholders.


• Strong leadership, communication, and stakeholder management skills.


• Experience with Palantir Foundry is a plus, but not required.


• Familiarity with CI/CD and infrastructure as code (e.g., Terraform, CloudFormation) is desirable.


• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.