GCP Data Engineer

Required Skills

• Strong experience in Python programming.


• Hands-on expertise with PySpark for large-scale data processing.


• Experience with Google Cloud Platform (GCP) services:


• BigQuery


• Dataproc


• Dataflow


• Cloud Storage


• Pub/Sub


• Cloud Composer (Airflow)


• Strong SQL skills and experience with relational databases.


• Experience building batch and real-time data pipelines.


• Good understanding of Data Lake and Data Warehouse concepts.


• Knowledge of distributed computing and Spark optimization techniques.


• Experience with Git, CI/CD pipelines, and Agile methodologies.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark and Python.
• Build and manage data processing solutions on Google Cloud Platform (GCP).
• Develop data ingestion frameworks from structured, semi-structured, and unstructured data sources.
• Implement data transformation, cleansing, and aggregation processes for analytics and reporting.
• Work with BigQuery, Cloud Storage, Dataproc, Dataflow, Pub/Sub, Composer (Airflow), and other GCP services.
• Optimize PySpark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Scientists, Analysts, and Application teams to deliver high-quality data solutions.
• Ensure data quality, governance, security, and compliance standards are met.
• Troubleshoot and resolve production data issues and performance bottlenecks.
• Participate in code reviews, CI/CD implementation, and DevOps practices.

referred Skills

• Experience with Kafka and streaming architectures.
• Knowledge of Terraform or Infrastructure as Code (IaC).
• Experience with Docker and Kubernetes.
• Understanding of data modeling and dimensional modeling concepts.
• Exposure to machine learning data pipelines.