Big Data Engineer

Are you passionate about helping people improve the impact of their business through the use of data and analytics? If you enjoy the challenges associated with making complex information systems more user-friendly and effective, we are looking for you to join a team of creative and highly motivated professionals who are driving innovation in the upstream oil field services industry! 

At National Oilwell Varco, we strive to lead technology innovation that delivers significant value to our customers. We are hiring a Big Data Engineer to help us build and support our data and analytics delivery pipeline.  

Responsibilities 

• Design, develop, and optimize scalable data ingestion, transformation, and analytics pipelines for IoT, operational, and business data using Spark and Delta Lake technologies. 

• Create and tune Spark transformation, aggregation, streaming, and machine learning workloads, including optimization of Spark clusters and processing infrastructure. 

• Develop reusable frameworks for ingesting, validating, enriching, and processing large-scale IoT and event-driven datasets. 

• Build and maintain analytical data models, curated datasets, and KPI calculation frameworks that enable product analytics, operational reporting, business intelligence, and self-service analytics. 

• Collaborate with Product, Engineering, Operations, and business stakeholders to define, implement, and maintain meaningful KPIs, metrics, and data products that support decision-making. 

• Design and implement real-time and batch processing solutions using event-based and streaming technologies. 

• Build and execute large-scale data migration solutions for transferring historical and operational data from source systems such as OSI PI and TimescaleDB into enterprise data platforms. 

• Develop monitoring, observability, reconciliation, and automation tools that ensure the reliability, quality, and performance of data pipelines, migrations, analytics workloads, and MLOps/DataOps processes. 

• Support the delivery of related platform capabilities, including APIs, data services, and integrations, while working within Agile and DevOps methodologies. 

• Document solution architecture, data lineage, business logic, KPI definitions, and operational procedures to support long-term maintainability and governance. 

Requirements

• Bachelor’s degree in computer science, information systems, or a related field. However, relevant experience will be considered. 

• A minimum of 5 years of relevant experience required.

• Advanced understanding with spark and/or Databricks(preferred), including spark cluster tuning. 

• Great understanding of distributed systems and partitioning. 

• Automation and scripting using .net, C#, Python, Javascript, GoLang, AWS Cloud APIs. 

• Database technologies and Timeseries databases like OSIPI, PostgreSQL, Timescale and SQL.. 

• Developing applications and/or scripts utilizing Timeseries data.

• Linux, Windows OS, Containers.

• Data management and data integration strategies.

• Amazon Web services and/or cloud technologies.

• Visualization/reporting tools such as Tableau, Spotfire, Grafana, etc. 

• Understanding of CI/CD practices and technologies. GitHub and GitHub Actions(preferred), Jenkins, TeamCity, Udeploy, etc. 

• Optimally communicating technical information.

• Working knowledge of ML and AI model development and the data science development life cycle is an added advantage.

• Experience with large datasets and understanding of stream and batch processing , Datadog, Brokers like Kafka or NATS , Container technologies, and terraform.