Big Data Engineer

Description

Are you passionate about helping people improve the impact of their business through the use of data and analytics? If you enjoy the challenges associated with making complex information systems more user-friendly and effective, we are looking for you to join a team of creative and highly motivated professionals who are driving innovation in the upstream oil field services industry!

At National Oilwell Varco, we strive to lead technology innovation that delivers significant value to our customers. We are hiring a Big Data Engineer to help us build and support our data and analytics delivery pipeline.

Responsibilities

This role carries a variety of responsibilities, including:

Create and tune spark data transformation and aggregation scripts for querying sensor data stored in delta lakehouse.
Tune spark clusters for running transformation and aggregation scripts, stream processing jobs and ML training jobs.
Designing and highly implementing performant data ingestion pipelines from multiple sources
Developing scalable and re-usable frameworks for ingesting of IoT datasets.
Working with event based / streaming technologies to ingest and process data
Working with other members of the project team to support delivery of additional project components (API interfaces)
Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.
Build large scale data migration scripts to migrate big data from native files in OSI PI and Timescale database into Data Lake
Use data migration tools such as TwinTalk to migrate data between platforms
Build utilities to monitor data migration scripts and streams
Build automation for repetitive tasks carried out by support team that facilitate ML operations.
Document the technical and usage details of all the tools