Azure Databricks Engineer

Plateau GRP is seeking a highly skilled Azure Databricks Engineer to support a large-scale data modernization initiative for the Federal Deposit Insurance Corporation (FDIC). The selected candidate will design, develop, and optimize enterprise-grade data lakehouse solutions using Azure Databricks, Apache Spark, and related Azure data services.

This position requires deep technical expertise in data engineering, advanced analytics enablement, and large-scale data pipeline optimization. The ideal candidate will collaborate with cross-functional teams to deliver secure, scalable, and high-performing data infrastructure within a modernized cloud environment.

Key Responsibilities

Data Pipeline Development & Management

• Design, build, and maintain scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and Apache Spark to process large volumes of structured and unstructured data.
• Integrate data from multiple sources using Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics.

Data Architecture & Optimization

• Architect and optimize data lakehouse solutions leveraging Delta Lake format and partitioning strategies.
• Implement data modeling best practices for efficient data storage, retrieval, and analytical performance.

Collaborative Analytics Platform Management

• Configure and manage Databricks workspaces, clusters, and compute resources.
• Implement security controls, access management, and cost optimization strategies aligned with FDIC standards.

Real-time & Batch Processing

• Develop streaming data solutions using Structured Streaming and implement batch processing jobs for transformation, aggregation, and integration.
• Ensure performance, scalability, and reliability of both real-time and batch data workflows.

MLOps & Data Science Support

• Collaborate with data scientists to deploy machine learning models and integrate them into data pipelines.
• Implement MLflow for model lifecycle management and automate ML pipelines for training and inference.

Performance Monitoring & Troubleshooting

• Monitor pipeline performance and system health across Databricks and Azure environments.
• Troubleshoot job failures, implement data quality checks, and optimize Spark jobs for improved efficiency and reduced processing cost.

Required Qualifications

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.
• Over 5 years of experience as a Data Engineer, with 3+ years specifically in Azure Databricks and related Azure data services.
• This position requires US Citizenship and the ability to pass/obtain any required government background check or security clearance. An active security is preferred.
• Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala.
• Hands-on experience designing, developing, and optimizing large-scale data pipelines in cloud environments.
• Technical proficiency with:
• Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics
• ETL/ELT processes, data modeling, and big data architectures
• Strong understanding of data governance, data security, and cloud operations in enterprise environments.
• Experience working within Agile delivery frameworks.

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer certification.
• Experience supporting federal data modernization or enterprise cloud transformation programs.
• Familiarity with DevOps/CI-CD pipelines, GitHub Enterprise, and Infrastructure as Code (Terraform).
• Experience supporting AI/ML initiatives and integrating data science workflows within Azure Databricks.

Plateau does not accept unsolicited resumes from third-party recruiters without a signed third-party agreement. Any unsolicited third-party resumes forwarded by recruiters to Plateau or to any of our managers or employees will be considered public information, may be treated as a direct application from the person identified in the resume, and will not be eligible for placement fee payment to the agency. 

EEO Statement: 

Plateau is an equal opportunity/affirmative action employer; all qualified applicants will receive consideration for employment without regard to age, gender, gender identification, sex, sexual orientation, color, race, creed, national origin, religion, marital status, parental status, citizenship status, ancestry, physical or mental disability, genetic information, veteran status, military status, or any other classification protected by federal, state, or local laws.

Plateau is committed to providing access and reasonable accommodations for individuals with disabilities. If you require reasonable accommodation, please email the Plateau HR Department at [email protected]  

Plateau is a proud Equal Opportunity Employer! (M/F/Disability/Vets)