Data Engineer III

Data Engineer is responsible for building a leading-edge Data & Analytics platform for enabling our new Troon Data Estate architecture & related analytics, business domain specific analytics & management (Finance, HR, Operations, & other), and general enterprise analytics needs. Designs, develops, maintains, and supports the cloud-based (Microsoft Azure) big data platform and uses modern data engineering design patterns and tools across the MSFT Data Platform.

Essential Duties:

Designs, builds and maintains scalable, automated data pipelines to enable Reporting, Data Visualization, Advanced Analytics, Data Science, and Machine Learning solutions.
Supports critical data pipelines with a scalable distributed architecture, including data ingestion (streaming, events, and batch), data integration (ETL, ELT, Azure Data Factory), and distributed data processing using Databricks Data/or other & Analytics and Azure Cloud Technology Stacks.
Builds cloud data solutions using multiple technologies, such as SQL, Python, Data Lake (Databricks Delta Lake/or other), Cloud Data Warehouse (Azure Synapse), RDBMS, NoSQL databases.
Implements best practices for data management in Microsoft Azure, such as data quality, data governance, data security, data lineage, data cataloging, etc.
Deploys, automates, maintains, and manages cloud-based production systems to ensure the availability, performance, scalability, and security of production systems.
Works with other engineers and stakeholders to understand business requirements and translate them into data models and architectures that meet performance, scalability, reliability, and security standards.
Owns end-to-end design and development, testing, the release of critical components using Databricks technology stack and Microsoft Azure cloud platforms and services.
Education/Experience: Minimum BA or BS degree in Computer Science, Information Systems, or related field required. MS in Business Analytics or related discipline preferred.
Minimum 6 years of experience in creating robust enterprise-grade data engineering pipelines using SQL, Python, Apache Spark, ETL, ELT, Databricks Technology Stack, Azure Cloud Services, Cloud-based Data and Analytics platforms required. 4-5 years preferred.
Experience with Microsoft Azure Data Services and SQL Server, tasks to include design, implementation, maintenance and performance monitoring of database instances.
Experience in distributed data (structured, semi-structured, unstructured, streaming) processing techniques using Apache Spark, Hadoop, Hive, Kafka, and big data ecosystem technologies preferred.
Job Knowledge, Skill, and Ability Preferences Knowledge and understanding of Microsoft Entra and RBAC models for data security and access controls a plus.
Emphasis on Azure SQL virtual machines (IasS) as well as Azure SQL Managed instances (PaaS), with a primary goal of providing consistent database availability and performance for all applications utilizing these environments.