Associate Principal

As an Associate Principal – Data Engineering, you’ll have an opportunity to work on the universe of data and solve some very interesting problems by creating and maintaining scalable data pipelines dealing with petabytes of data. All our projects entail working on cutting edge technologies, petabyte scale data processing systems, data warehouses and data lakes to help manage the ever-growing information needs of our customers.

The responsibilities are detailed as below:

• Experience in understanding and translating data, analytic requirements and functional needs into technical requirements while working with global customers

• Build and maintain data pipelines to support large scale data management in alignment with data strategy and data processing standards

• Strong experience in database, data warehouse & data lake design & architecture

• Able to train & mentor team members

• Experience in Database programming using multiple flavor of SQL

• Deploy scalable data pipelines for analytical needs

• Experience in Big Data ecosystem - on-prem (Hortonworks/MapR) or Cloud (Dataproc/EMR/HDInsight)

• Worked on query languages/tools such as Hadoop, Pig, SQL, Hive, Sqoop and SparkSQL.

• Experience in any orchestration tool such as Airflow/Oozie for scheduling pipelines

• Exposure to latest cloud ETL tools such as Glue/ADF/Dataflow

• Understand and execute IN memory distributed computing frameworks like Spark (and/or DataBricks) and its parameter tuning, writing optimized queries in Spark

• Hands-on experience in using Spark Streaming, Kafka and Hbase

• Experience in any orchestration tool such as Airflow/Oozie

• Experience working in an Agile/Scrum development process

REQUIRED QUALIFICATIONS

We are looking for individuals who are curious, excited about learning, and navigating through the uncertainties and complexities that are associated with growing a company. Some qualifications that we think would help you thrive in this role are:

•BE/BS/MTech/MS in computer science or equivalent work experience.

• 10+ years of experience in building data processing applications using Hadoop, Spark and NoSQL DB and Hadoop streaming.

• 5+ years of experience in building scalable enterprise data warehousing and modelling on prem and/or cloud (AWS, GCP, Azure)