Data Integration Engineer

Mandatory Skills

Databricks & Python

Job Description

Core Technical Skills

• Python - Advanced level (pandas, requests, boto3 libraries)

• API Integration - REST APIs, authentication (OAuth, API keys), error handling

• Cloud Platforms - AWS (S3, EC2, IAM) proficiency

• Databricks/Spark - PySpark, Delta Lake, SQL optimization

• Data Pipeline Tools - Airflow, Databricks Workflows, or equivalent orchestration

• SQL - Complex queries, window functions, performance tuning

Security & Enterprise Knowledge

• Understanding of security telemetry (logs, alerts, events, metrics)

• Familiarity with at least 2-3 security platforms (JAMF, Intune, Qualys, CrowdStrike,

Okta)

• Experience with AWS CloudTrail or similar cloud audit logs

• Knowledge of data security and compliance (encryption, access controls)

Additional Competencies

• Version control (Git)

• CI/CD pipeline familiarity

• Data quality and validation frameworks

• Performance tuning and optimization mindset

Education Qualificaiton

BE, BTech, MCA

Roles & Responsibilities

Data Integration & Connectivity

• Design and build connectors between enterprise security platforms (JAMF, Intune, Qualys,

CrowdStrike, Okta, AWS CloudTrail) and Databricks

• Establish secure API connections and authentication mechanisms with each source system

• Implement data extraction logic using REST APIs, SDKs, and native connectors

• Handle varying data formats and transformation requirements across heterogeneous sources

Data Pipeline Development

• Develop automated Python scripts for continuous data ingestion

• Build orchestration workflows using Apache Airflow, Databricks Workflows, or similar tools

• Implement error handling, retry mechanisms, and monitoring for pipeline reliability

• Ensure near real-time or scheduled data synchronization based on requirements

AWS S3 & Data Staging

• Configure S3 bucket architecture and data partitioning strategy

• Land raw security telemetry data in S3 with proper folder structures

• Implement lifecycle policies and data retention strategies

• Ensure proper encryption and access controls

Databricks Platform Configuration

• Create and optimize Databricks tables