Data Engineer (Big Data, Spark, Hadoop, Python) – Contract - Part-Time

Job Description

• Design, develop, optimize, and maintain data architecture and pipelines that adhere to ETL principles and business goals.

• Solve complex data problems to deliver insights that help the organization achieve its goals.

• Code in Python with tools like Apache Spark to build a multi-cluster data warehouse.

• Interact with other technology teams to define, prioritize, and ensure smooth deployments for other operational components.

• Advise, consult, mentor, and coach other data and analytics professionals on data standards and practices.

• Foster a culture of sharing, reuse, design for scale stability, and operational efficiency of data and analytical solutions.

• Codify best practices for future reuse in the form of accessible, reusable patterns, templates, and code bases to facilitate data capturing and management.

• Strong experience in mapping attributes, data profiling , data cleansing , and technical data quality etc.

• Strong experience in Ansi SQL and in-depth knowledge with structured , semi-structured & unstructured data

• Must have good knowledge of data lake and working experience of migration projects in cloud with providers like AWS or Microsoft AZURE or GCP

• Good to have experience in working with No SQL, Spark SQL using AWS Glue, EMR, and columnar data store.

• Good to understand data security features like data masking, data encryption , role based & fine grain access control mechanisms etc.

Qualifications

• 4+ years of relevant experience in data engineering/analytics space

• Expertise in SQL and data analysis and strong hands-on expertise with at least one programming language: Python.

• Strong knowledge in one or more of the following big data tools: Hive, Hadoop Impala, Spark, Kafka

• Strong expertise in ETL, reporting tools, data governance, data warehousing, and hands-on experience.

• Experience developing solutions for cloud computing services and infrastructure.

• Experience developing and maintaining data warehouses in big data solutions.

• Up to date on industry trends within the analytics space from a data acquisition processing, engineering, and management perspective

• Experience in agile development.

• Strong people skills, specifically in collaboration and teamwork

• High level of curiosity, creativity, and problem-solving capabilities