Sr. Data Engineer - Join a Market Leader

Job Overview: As a Senior Data Engineer, you will be at the forefront of shaping and executing our data engineering strategies. You will lead a team of skilled engineers, collaborate across departments, and leverage your proficiency in Python, AWS and Apache Spark to deliver high-performance data solutions that drive insights and innovation.

Responsibilities:

1. Data Pipeline Architecture and Development:

o Architect, design, and implement scalable and robust data pipelines for collecting, processing, and transforming data from diverse sources, including IoT devices.

o Optimize data pipelines for performance, reliability, and scalability.

o Collaborate with data scientists and analysts to ensure data availability and accuracy.

2. Python, Scala, and ETL Development:

o Utilize your strong Python and Spark programming skills to develop and maintain ETL processes, data integration, and data transformation workflows.

o Create and enhance data processing scripts and programs to handle large datasets.

3. AWS and Cloud Infrastructure:

o Leverage AWS services such as Lambda, Glue, Kinesis, AWS SQS/SNS, AWS DynamoDB, S3, and Athena to design and deploy cloud-based data solutions.

o Design, implement, and manage data storage, processing, and analytics solutions in AWS.

4. Apache Flink/Apache Spark Expertise:

o Utilize your expertise in Apache Flink/ Apache Spark to develop real-time data processing pipelines, ensuring low-latency and high-throughput data streaming.

5. Experience in Snowflake

6. Expert in SQL and SQL Administration.

7. Experience in Denodo virtualization tool

8. Data Quality and Governance:

o Implement rigorous data quality checks, validation processes, and data governance best practices.

o Continuously monitor and maintain data pipelines to ensure data accuracy and consistency.

9. Documentation and Communication:

o Thoroughly document data engineering processes, pipeline architectures, and technical specifications.

Cloud BC Labs Inc is a digital transformation organization aimed at creating seamless solutions for clients to effectively manage their business operations. The company specializes in Business and Management Consulting, AI/ML, Data Analytics & Visualization, Cloud Data Warehouse Migration, Snowflake Implementation, Informatica Implementation & Upgrade, Staffing Services and Data Management Solutions