Data Engineer

Job Overview

Nextlink is seeking for a Data Engineer with strong expertise in Palantir Foundry, Python, and PySpark to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience with Foundry Ontology, data pipelines, modular applications, and end-to-end data solution implementation, along with a solid understanding of data engineering principles and distributed computing.

Key Responsibilities

1. Palantir Foundry Platform Expertise

• Develop and maintain data pipelines, Foundry Ontology, and modular applications within the Palantir Foundry platform.
  
• Design and implement end-to-end data solutions, from data ingestion and transformation to data modeling and consumption.
  
• Integrate data from multiple source systems into Palantir Foundry.
  
• Build and maintain scalable, reusable data workflows aligned with business requirements.
  
• Leverage Foundry capabilities to support data-driven applications and analytics.
  

2. Programming and Scripting – Python & PySpark

• Develop Python and PySpark scripts for data processing, transformation, cleansing, and automation.
  
• Build reusable Python functions and libraries to extend Palantir Foundry capabilities.
  
• Develop modular and maintainable code for ETL workflows and data transformation processes.
  
• Process large datasets efficiently using Python libraries such as Pandas and NumPy, along with PySpark for distributed processing.
  
• Optimize code for performance, scalability, and reliability.
  
• Use SQL for data querying, validation, and data modeling, as required.
  

3. Data Engineering Fundamentals

• Design, develop, and maintain ETL/ELT pipelines for data ingestion and transformation.
  
• Build efficient data models and optimize data processing workflows.
  
• Work with structured and unstructured data from multiple sources.
  
• Implement data quality checks, validation, and error-handling mechanisms.
  
• Ensure data consistency, reliability, and availability across pipelines.
  

4. Big Data and Distributed Systems

• Develop scalable data processing solutions using Apache Spark and PySpark.
  
• Apply distributed computing concepts to handle large-scale data processing requirements.
  
• Troubleshoot pipeline failures and resolve data processing issues.
  
• Perform performance tuning and optimization of Spark jobs and data workflows.
  
• Follow best practices for scalable and efficient data engineering.
  

Requirements

Essential Skills

• Hands-on experience with Palantir Foundry and its core capabilities.
  
• Strong programming skills in Python and PySpark.
  
• Experience developing and maintaining ETL/ELT pipelines.
  
• Knowledge of Foundry Ontology, data pipelines, and modular applications.
  
• Strong understanding of data transformation, data modeling, and data integration.
  
• Experience with Apache Spark and distributed data processing.
  
• Understanding of performance optimization and scalable data processing techniques.
  

Desirable Skills

• Familiarity with SQL for querying and data modeling.
  
• Experience with Pandas and NumPy for data manipulation and analysis.
  
• Knowledge of data quality, validation, and pipeline monitoring practices.
  
• Experience integrating data from multiple source systems.
  
• Understanding of data governance and enterprise data engineering best practices.
  

Keywords

Palantir Foundry, Palantir Core, PySpark, Python, Apache Spark, Data Engineering, ETL/ELT, Foundry Ontology, Data Pipelines, Modular Applications, SQL, Data Transformation, Data Modeling, Distributed Computing, Performance Optimization.

Primary Skills: Digital – PySpark, Palantir Core

Role Focus: Palantir Foundry development, Python/PySpark-based data engineering, and end-to-end data pipeline implementation.