Design, develop, test, deploy and monitor data pipelines in Databricks on AWS from a wide variety of data sources.
Design, develop, test, deploy and monitor scalable code with PySpark and SQL in Databricks.
Identify opportunities to improve internal process through code optimisation and automation.
Build data quality dashboards, lineage flows / and or monitoring tools to utilize the data pipeline, providing active monitoring and actionable insight into overall data quality and data governance.
Assist in migrating data from legacy systems onto newly developed solutions.
Follow and lead best practices on all data security, retention, and privacy policies.
Requirements:
5+ years experience of building ETL/ELT pipelines.
Proven competency in solution design, development, implementation, reporting and analysis.
Proficiency in Apache-Spark, Python and SQL languages.
Proficiency in working with Text, Delta, Parquet, JSON, CSV, and XML data formats.
Working knowledge of Spark structured streaming.
AWS infrastructure experience, specifically working with S3.
Solid understanding of git-based version control, DevOps, and CI/CD. Experience of working on Atlassian stack a plus.
Knowledge of common web API frameworks and web services.
Strong teamwork, relationship, and client management skills, and the ability to influence peers and senior management to accomplish team goals.
Willingness to embrace modern technology, best practice, and ways of work.
All Job Ads are subject to GrabJobs’s Terms of Service. We allow users to flag postings that may be in violation of those terms. Job Ads may also be flagged by GrabJobs moderation team. However, no moderation system is perfect, and flagging a posting does not ensure that it will be removed.
Be the first to receive the latest Others Full-Time Jobs in Vietnam.
Setup your job alert:
By activating job alerts, I agree to GrabJobs Terms & Privacy Policy. I can unsubscribe to job alerts anytime.
Skip
GrabJobs is the no1 job portal in Vietnam, connecting you to thousands of jobs fast!
Find the best jobs in Vietnam, apply in 1 click and get a job today!