Senior Data Engineer

JOB DESCRIPTION

JOB TITLE: Senior Data Engineer

DEPARTMENT: 500 – IT

REPORTS TO: Chief of Information Officer

JOB SUMMARY: 

The Senior Data Engineer – Databricks is responsible for building and evolving PharmaLink’s enterprise data, analytics, and AI platform. This hands-on role spans the modern Databricks ecosystem, including data engineering, lakehouse architecture, analytics and dashboards, AI, Databricks Apps, and Lakebase. The role requires strong engineering fundamentals, independent problem-solving, technical ownership, and the ability to design, build, and operate reliable, scalable production solutions.

A hands-on builder approach is essential, with a focus on developing solutions rather than simply administering platforms. The ideal candidate can work across data engineering, platform architecture, analytics, AI, and data applications; independently investigate complex problems; and take solutions from initial design through production.

KEY RESPONSIBILITIES:

• Design, build, and operate enterprise data solutions on Databricks. 
• Develop scalable ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake. 
• Design curated data models and reusable data products for analytics, applications, and AI. 
• Implement Unity Catalog, security, lineage, access controls, and data-governance capabilities. 
• Build and optimize Databricks SQL Warehouses and AI/BI Dashboards. 
• Develop data and AI applications using Databricks Apps. 
• Design and implement appropriate solutions using Databricks Lakebase. 
• Enable AI/ML and Generative AI use cases using Databricks capabilities. 
• Build automated data-quality, reconciliation, anomaly-detection, and monitoring processes. 
• Integrate data from databases, SaaS platforms, APIs, files, and other enterprise systems. 
• Optimize Spark workloads, SQL queries, pipelines, performance, and platform cost. 
• Establish engineering standards for Git, CI/CD, testing, monitoring, and production deployment. 
• Troubleshoot complex production and data issues and perform root-cause analysis. 
• Partner with Data Governance, BI, application, and business teams to deliver trusted data products. 

QUALIFICATIONS:

• 8+ years of experience in data engineering, data platforms, or related engineering roles. 
• 4+ years of hands-on Databricks experience in production environments. 
• Advanced SQL, Python, PySpark, and Apache Spark skills. 
• Strong experience with Delta Lake and Unity Catalog. 
• Strong ETL/ELT, data modeling, and enterprise data architecture experience. 
• Experience with Databricks SQL and AI/BI Dashboards. 
• Familiarity with Databricks capabilities across AI/ML, Generative AI, Databricks Apps, and Lakebase. 
• Strong relational database and data-engineering fundamentals. 
• Experience with Git, CI/CD, automated testing, monitoring, and production support. 
• Experience working with cloud-based enterprise data platforms. 
• Must maintain a positive attitude in a high-performance team environment.
• Must pass background check and random drug screenings.
• Must be adaptable and flexible in dealing with a diverse work force

PREFERRED QUALIFICATIONS:

• Databricks on Microsoft Azure. 
• Azure Data Factory. 
• Microsoft SQL Server. 
• Fivetran or similar CDC/ELT technologies. 
• Power BI. 
• Databricks Workflows. 
• MLflow and AI/model lifecycle capabilities. 
• API and application development experience. 
• Azure DevOps or GitHub-based CI/CD. 
• Databricks professional certification. 
• Experience in healthcare, life sciences, or another regulated industry. 

PHYSICAL REQUIREMENTS:

• Requires light physical activity performing non-strenuous daily activities of an administrative nature (i.e., sustained phone, headset, and keyboard use) as well as periods of prolonged sitting.