Lead Data Engineer

About PrePass
PrePass® is North America's most utilized and technologically advanced weigh station bypass and toll payment platform. Proven PrePass technologies enable safe, qualified motor carriers to bypass inspection facilities at highway speeds, saving them time, fuel, and money while reducing emissions. As the only provider to offer bypass and tolling solutions, PrePass technology allows fleets to regain control of toll costs, eliminate toll violations, and automatically resolve max toll disputes. PrePass is the only preclearance system developed, owned, and operated in the United States of America as well as the American Trucking Associations’ only Endorsed Corporate Partner. That’s why more than 105,000 fleets subscribe over 750,000 commercial vehicles to PrePass services.
Position Description
As a Lead Data Engineer, you will be at the forefront of pioneering solutions that cater to the evolving demands of our growing business. In this capacity, you will take the lead in formulating the design, development, and implementation of our next-generation, cloud-native data architecture. Additionally, you will mentor and guide a team of talented data engineers. This position demands a blend of technical acumen and strategic thinking. We’re looking for a trailblazer who thrives on innovation and is committed to continuous learning. You should be equally adept at coding as you are in conceptualizing and communicating complex solutions across various teams and domains.
Your Key Responsibilities
Design and implement highly scalable, distributed architecture using Azure Databricks, Delta Lake, and other Microsoft Azure services.
Act as a technical expert, lead projects, and optimize production systems to ensure reliability, performance, scalability, and security.
Partner with Product Management and Engineering leadership to architect and build solutions that deliver on roadmap features.
Implement best practices in Microsoft Azure for data quality, data governance, data security, data lineage, and data cataloging.
Guide and mentor less experienced engineers.
Oversee and provide leadership to a team of approximately 5 engineers.
Actively monitor workflows and compute to identify refinements that will improve efficiency and reduce costs.
Build comprehensive solutions for orchestration and observation.
Troubleshoot system issues, prioritize fixes, and distribute work among the team.
Prepare, manage, and supervise code releases to Production.
Spearhead prototyping exercises and demonstrate value and feasibility of the solutions and architecture.
Requirements
Qualifications
Required
Bachelor’s degree in computer science, related technical field, or equivalent experience.
Minimum 8 years of overall experience in designing and building large scale database solutions, with at least 2 years in a leadership role.
Minimum 3 years programming experience in Python and Pyspark.
Expert SQL skills.
Extensive experience building ETL/ELT solutions using Azure Databricks, Azure Data Factory, or similar tools.
Experience designing data structures and models for data lakes.
Thorough understanding of distributed storage and distributed computing.
Proficiency with reporting and data visualization tools, specifically PowerBI.
Proficiency with SSIS.
Versed in software production engineering practices, version control, code peer reviews, automated testing, and CI/CD.
Exceptional collaborative abilities, with a talent for navigating fluid environments and embracing change.
A commitment to staying abreast of emerging technologies and industry best-practices to drive continuous improvement in development methodologies.
Preferred
Understanding of ML and AI concepts, algorithms, and techniques.
Experience with both batch and streaming data sources.
Familiarity with No-SQL databases.
Familiarity of event driven architecture – queues, batches, and pub/sub models.
Experience working with data from enterprise systems like ERPs and CRMs.
Familiarity with DevOps CI/CD implementation methodologies.
Benefits
How We Will Take Care of You
Robust benefit package that includes medical, dental, and vision that start on date of hire.
Paid Time Off, to include vacation, sick, holidays, and floating holidays.
401(k) plan with employer match.
Company-funded “lifestyle account” upon date of hire for you to apply toward your physical and mental well-being (i.e., ski passes, retreats, gym memberships).
Tuition Reimbursement Program.
Employee Assistance Program (available at no cost to you).
Voluntary benefits, to include but not limited to Legal and Pet Discounts.
Company-sponsored and funded “Culture Team” that focuses on the Physical, Mental, Professional well-being of employees.
Community Give-Back initiatives.
Culture that focuses on Employee Development initiatives.