Databricks Practice Lead / Engineering Manager

Position Summary

Scicom Infrastructure Services is seeking an experienced Databricks Practice Lead / Engineering Manager to provide hands-on technical leadership while managing a team of data engineers, architects, and consultants supporting complex enterprise and government programs.

This role requires a senior Databricks expert who can design and oversee modern data platforms, establish technical standards, guide delivery teams, and remain actively involved in architecture, troubleshooting, code reviews, and client-facing solution development. The successful candidate will balance deep technical expertise with strong people leadership, delivery management, and stakeholder communication skills.

Key Responsibilities

Databricks Technical Leadership

• Serve as the organization’s subject-matter expert for the Databricks Lakehouse Platform.


• Design scalable, secure, and highly available data architectures using Databricks, Apache Spark, Delta Lake, and cloud-native technologies.


• Lead the implementation of batch, streaming, ETL, ELT, analytics, machine-learning, and AI-enabled data solutions.


• Define architectural standards for medallion architectures, data modeling, ingestion, transformation, orchestration, and data consumption.


• Establish governance frameworks using Unity Catalog, including data lineage, access controls, auditing, metadata management, and secure data sharing.


• Guide Databricks workspace design, cluster configuration, serverless computing, workload isolation, performance tuning, and cost optimization.


• Oversee integration between Databricks and cloud platforms such as Microsoft Azure, AWS, or Google Cloud.


• Develop or review solutions involving PySpark, Spark SQL, Python, Delta Live Tables, Structured Streaming, Auto Loader, MLflow, and Databricks Workflows.


• Lead platform migrations and modernization efforts from legacy databases, data warehouses, Hadoop environments, and traditional ETL platforms.


• Establish development standards for source control, automated testing, CI/CD, infrastructure as code, monitoring, and production support.


• Conduct architecture reviews, code reviews, technical assessments, and root-cause analyses.


• Evaluate emerging Databricks capabilities and recommend appropriate adoption strategies.

Team Leadership and Management

• Manage, mentor, and develop a team of Databricks engineers, data engineers, architects, and technical consultants.


• Assign resources and responsibilities based on project needs, employee strengths, availability, and technical complexity.


• Establish measurable goals, performance expectations, development plans, and technical competency standards.


• Conduct regular one-on-one meetings, performance reviews, coaching sessions, and technical development activities.


• Support recruiting, interviewing, candidate evaluation, onboarding, and workforce planning.


• Identify technical or performance gaps and coordinate training, mentoring, or corrective action as appropriate.


• Promote collaboration, accountability, documentation, knowledge sharing, and continuous improvement.


• Develop reusable accelerators, reference architectures, templates, and delivery playbooks.


• Build and maintain a strong Databricks practice capable of supporting multiple concurrent client engagements.

Program and Delivery Management

• Provide delivery oversight for Databricks and data-engineering projects from planning through implementation and operational support.


• Translate business, functional, security, and contractual requirements into technical plans and deliverables.


• Develop project estimates, staffing plans, delivery schedules, milestones, and risk-mitigation strategies.


• Monitor project scope, schedule, quality, budget, resource utilization, dependencies, and technical risks.


• Ensure deliverables meet client requirements, internal quality standards, security controls, and contractual commitments.


• Coordinate work across engineering, cloud, cybersecurity, data governance, analytics, project-management, and client teams.


• Track delivery metrics and provide clear status reports to internal leadership, clients, and program stakeholders.


• Lead technical escalations and ensure issues are resolved promptly and appropriately documented.


• Support statements of work, technical proposals, solution estimates, presentations, and client demonstrations.


• Participate in client meetings as the technical and delivery authority for Databricks-related work.

Required Qualifications

• Bachelor’s degree in computer science, information technology, data engineering, engineering, or a related discipline.


• At least 10 years of experience in data engineering, data architecture, analytics engineering, or related technology roles.


• At least 5 years of hands-on experience designing and implementing solutions using Databricks.


• At least 3 years of experience managing or formally leading technical engineering teams.


• Advanced experience with:
  
  
  
  • Databricks Lakehouse Platform
  
  
  • Apache Spark and PySpark
  
  
  • Spark SQL and advanced SQL development
  
  
  • Delta Lake and medallion architecture
  
  
  • Unity Catalog and enterprise data governance
  
  
  • ETL and ELT pipeline architecture
  
  
  • Batch and real-time data processing
  
  
  • Data modeling and data warehousing
  
  
  • Python-based data engineering
  
  
  • Databricks Workflows, Jobs, and cluster management
  
  
  
  


• Experience deploying Databricks solutions in Azure, AWS, or Google Cloud.


• Experience with CI/CD, Git-based development, automated testing, and infrastructure as code.


• Demonstrated ability to optimize Spark workloads, cluster configurations, query performance, reliability, and cloud costs.


• Experience managing technical delivery, resource assignments, risks, schedules, and client expectations.


• Strong written, verbal, presentation, documentation, and stakeholder-management skills.


• Ability to explain complex technical concepts to executives, business stakeholders, and nontechnical audiences.

Preferred Qualifications

• Databricks Certified Data Engineer Professional, Databricks Certified Data Engineer Associate, or Databricks Certified Machine Learning Professional.


• Databricks Certified Data Architect or comparable advanced architecture credentials.


• Microsoft Azure, AWS, or Google Cloud professional-level certification.


• Experience working in a consulting, professional-services, systems-integration, or managed-services environment.


• Experience supporting federal, state, or local government clients.


• Experience working with major consulting or systems-integration partners.


• Knowledge of federal security, privacy, governance, and compliance requirements.


• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Terraform.


• Experience with MLflow, MLOps, generative AI, Databricks Mosaic AI, vector search, or machine-learning deployment.


• Familiarity with data standards, metadata frameworks, data catalogs, data-sharing protocols, and open-data environments.


• Experience managing geographically distributed or remote technical teams.


• Experience contributing to proposals, technical responses, statements of work, and project estimates.

Leadership Competencies

The successful candidate will demonstrate:

• Hands-on technical credibility and sound architectural judgment.


• The ability to lead without becoming disconnected from the technology.


• Strong accountability for team performance and project outcomes.


• Effective coaching, delegation, and conflict-resolution skills.


• Clear and proactive communication with clients and internal leadership.


• The ability to manage competing priorities in a fast-paced consulting environment.


• A commitment to quality, security, documentation, and continuous improvement.

Success Measures

Performance in this role will be evaluated based on:

• Quality, scalability, security, and reliability of Databricks solutions.


• On-time and within-budget delivery of client commitments.


• Team performance, retention, development, and technical growth.


• Client satisfaction and effective stakeholder communication.


• Reduction in delivery risks, production incidents, and technical debt.


• Adoption of standardized architectures, engineering practices, and reusable solutions.


• Effective management of Databricks consumption, infrastructure, and cloud costs.


• Growth and maturity of the organization’s Databricks practice.